2023-12-16T10:56:59.302 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-16T10:56:59.302 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988 --name yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} -- /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/orig.config.yaml 2023-12-16T10:56:59.350 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T10:56:59.451 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988 branch: main description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: yweinste@redhat.com first_in_suite: false job_id: '7493988' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2970 sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPkeJ1senFIzDxS3+j0Z32wtuNeGa/Dm2hwzGwxTXgWXVMGTT5Ri8kCA/tcuyDVlebct8GeVWwqpOqLgBW1Myw4= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjmBWJ/0sPWKHpp3zLV72klfKcofmL77fVFaUFRdBAiT3e1l9nCNfxDLV1niDNP2m9XRBvQQUBkDRhoK3L5IRQ= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-16_00:13:26 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-16T10:56:59.452 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa; will attempt to use it 2023-12-16T10:56:59.452 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks 2023-12-16T10:56:59.452 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-16T10:56:59.454 INFO:teuthology.task.internal:Checking packages... 2023-12-16T10:56:59.481 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6d933094a156ec7587742e5aa25fba5ba27ec678' 2023-12-16T10:56:59.481 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T10:56:59.481 INFO:teuthology.packaging:ref: None 2023-12-16T10:56:59.481 INFO:teuthology.packaging:tag: None 2023-12-16T10:56:59.481 INFO:teuthology.packaging:branch: main 2023-12-16T10:56:59.481 INFO:teuthology.packaging:sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T10:56:59.482 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2023-12-16T10:56:59.715 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-196.g85fe3c65 2023-12-16T10:56:59.717 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-16T10:56:59.723 INFO:teuthology.task.internal:no buildpackages task found 2023-12-16T10:56:59.723 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-16T10:56:59.757 INFO:teuthology.task.internal:Saving configuration 2023-12-16T10:56:59.777 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-16T10:56:59.785 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T10:56:59.819 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 10:51:46.992645', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPkeJ1senFIzDxS3+j0Z32wtuNeGa/Dm2hwzGwxTXgWXVMGTT5Ri8kCA/tcuyDVlebct8GeVWwqpOqLgBW1Myw4='} 2023-12-16T10:56:59.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 10:51:46.991003', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjmBWJ/0sPWKHpp3zLV72klfKcofmL77fVFaUFRdBAiT3e1l9nCNfxDLV1niDNP2m9XRBvQQUBkDRhoK3L5IRQ='} 2023-12-16T10:56:59.850 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-16T10:56:59.857 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-16T10:56:59.857 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-16T10:56:59.857 INFO:teuthology.run_tasks:Running task console_log... 2023-12-16T10:56:59.961 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1b58695c10>, signals=[15]) 2023-12-16T10:56:59.962 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-16T10:56:59.970 INFO:teuthology.task.internal:Opening connections... 2023-12-16T10:56:59.970 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T10:56:59.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T10:57:00.063 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T10:57:00.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T10:57:00.180 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-16T10:57:00.189 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2023-12-16T10:57:00.207 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2023-12-16T10:57:00.207 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2023-12-16T10:57:00.263 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T10:57:00.264 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T10:57:00.265 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2023-12-16T10:57:00.298 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2023-12-16T10:57:00.316 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2023-12-16T10:57:00.316 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T10:57:00.373 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T10:57:00.374 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T10:57:00.374 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2023-12-16T10:57:00.397 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-16T10:57:00.412 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-16T10:57:00.436 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-16T10:57:00.436 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2023-12-16T10:57:00.439 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2023-12-16T10:57:00.454 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-16T10:57:00.464 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-16T10:57:00.465 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2023-12-16T10:57:00.495 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2023-12-16T10:57:00.521 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-16T10:57:00.599 INFO:teuthology.run_tasks:Running task kernel... 2023-12-16T10:57:00.615 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-16T10:57:00.615 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-16T10:57:00.616 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T10:57:00.616 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T10:57:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T10:57:00.632 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-12-16T10:57:00.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T10:57:00.633 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2023-12-16T10:57:00.689 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-361.el9.x86_64 2023-12-16T10:57:00.689 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.14.0-361.el9.x86_64 2023-12-16T10:57:00.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2023-12-16T10:57:00.690 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-361.el9.x86_64 2023-12-16T10:57:00.690 INFO:teuthology.task.kernel:Running kernel on smithi087: 5.14.0-361.el9.x86_64 2023-12-16T10:57:00.690 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2023-12-16T10:57:02.727 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-16T10:57:02.727 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-16T10:57:04.045 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-12-16T10:57:04.069 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-12-16T10:57:04.851 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 7.9 MB 00:00 2023-12-16T10:57:08.743 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 7.9 MB 00:04 2023-12-16T10:57:10.935 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 18 MB 00:05 2023-12-16T10:57:11.504 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-12-16T10:57:16.145 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 5.8 MB 00:01 2023-12-16T10:57:16.444 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 5.8 MB 00:02 2023-12-16T10:57:17.570 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2023-12-16T10:57:18.053 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-12-16T10:57:19.316 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2023-12-16T10:57:20.006 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-16T10:57:24.213 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-12-16T10:57:24.762 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-16T10:57:26.303 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T10:57:26.304 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T10:57:26.395 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T10:57:26.399 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T10:57:26.399 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-12-16T10:57:26.399 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T10:57:26.399 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T10:57:26.399 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T10:57:26.399 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout:Install 4 Packages 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T10:57:26.400 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 96 M 2023-12-16T10:57:26.401 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 121 M 2023-12-16T10:57:26.401 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T10:57:26.902 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T10:57:26.902 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T10:57:26.994 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T10:57:27.001 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T10:57:27.001 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-16T10:57:27.001 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T10:57:27.001 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T10:57:27.001 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T10:57:27.001 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T10:57:27.001 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 96 M 2023-12-16T10:57:27.002 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 121 M 2023-12-16T10:57:27.003 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T10:57:27.422 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 18 MB/s | 5.6 MB 00:00 2023-12-16T10:57:27.730 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 33 MB/s | 20 MB 00:00 2023-12-16T10:57:27.798 INFO:teuthology.orchestra.run.smithi087.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 5.0 MB/s | 5.6 MB 00:01 2023-12-16T10:57:28.313 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 32 MB/s | 38 MB 00:01 2023-12-16T10:57:28.390 INFO:teuthology.orchestra.run.smithi087.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 12 MB/s | 20 MB 00:01 2023-12-16T10:57:28.622 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 27 MB/s | 32 MB 00:01 2023-12-16T10:57:28.623 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T10:57:28.623 INFO:teuthology.orchestra.run.smithi160.stdout:Total 59 MB/s | 96 MB 00:01 2023-12-16T10:57:29.190 INFO:teuthology.orchestra.run.smithi087.stdout:(3/4): kernel-modules-core-5.14.0-391.el9.x86_6 23 MB/s | 32 MB 00:01 2023-12-16T10:57:29.478 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T10:57:29.524 INFO:teuthology.orchestra.run.smithi087.stdout:(4/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 13 MB/s | 38 MB 00:02 2023-12-16T10:57:29.527 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T10:57:29.527 INFO:teuthology.orchestra.run.smithi087.stdout:Total 31 MB/s | 96 MB 00:03 2023-12-16T10:57:29.853 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T10:57:29.853 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T10:57:30.386 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T10:57:30.757 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T10:57:30.757 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T10:57:31.178 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T10:57:31.178 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T10:57:32.092 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T10:57:32.092 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T10:57:33.828 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T10:57:34.478 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T10:57:34.606 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T10:57:35.002 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T10:57:35.636 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T10:57:35.766 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T10:57:37.414 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T10:57:37.609 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T10:57:38.519 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T10:57:38.703 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T10:57:42.278 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T10:57:42.399 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:57:43.308 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T10:57:43.495 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:57:47.039 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:57:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:16.352 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:16.395 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:16.826 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:16.826 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T10:58:16.826 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T10:58:16.826 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T10:58:16.925 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:16.968 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:17.203 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:17.203 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T10:58:17.203 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T10:58:17.203 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.203 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.203 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.203 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.204 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T10:58:17.204 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T10:58:17.390 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:17.390 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T10:58:17.390 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T10:58:17.390 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T10:58:17.690 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T10:58:17.807 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T10:58:17.808 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T10:58:17.842 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.842 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T10:58:17.843 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T10:58:17.843 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T10:58:17.843 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T10:58:17.843 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-16T10:58:17.843 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T10:58:17.843 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T10:58:17.962 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T10:58:17.962 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T10:58:17.962 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2023-12-16T10:58:17.997 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2023-12-16T10:58:17.997 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2023-12-16T10:58:17.997 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.06 2023-12-16T10:58:17.997 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 68.el9 2023-12-16T10:58:17.997 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2023-12-16T10:58:17.997 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Group : Unspecified 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 8264254 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : builder@centos.org 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T10:58:17.998 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T10:58:17.999 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T10:58:17.999 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T10:58:17.999 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2023-12-16T10:58:17.999 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T10:58:17.999 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T10:58:18.000 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T10:58:18.000 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 5.14.0-391.el9.x86_64 2023-12-16T10:58:18.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T10:58:18.295 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T10:58:18.432 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T10:58:18.433 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T10:58:18.433 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T10:58:18.433 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T10:58:18.433 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T10:58:18.433 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-16T10:58:18.433 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T10:58:18.433 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T10:58:18.552 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T10:58:18.553 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T10:58:18.553 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.06 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 68.el9 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Group : Unspecified 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 8264254 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T10:58:18.587 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : builder@centos.org 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T10:58:18.588 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T10:58:18.589 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T10:58:18.589 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 5.14.0-391.el9.x86_64 2023-12-16T10:58:18.589 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T10:58:18.646 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2023-12-16T10:58:19.209 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2023-12-16T10:58:20.113 INFO:teuthology.orchestra.run.smithi160.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T10:58:20.154 INFO:teuthology.orchestra.run.smithi160.stderr:done 2023-12-16T10:58:20.156 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2023-12-16T10:58:20.171 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.VhQcXDHWBR 2023-12-16T10:58:20.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VhQcXDHWBR 2023-12-16T10:58:20.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.VhQcXDHWBR 2023-12-16T10:58:20.385 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.VhQcXDHWBR is 6KB 2023-12-16T10:58:20.434 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.VhQcXDHWBR 2023-12-16T10:58:20.448 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T10:58:20.514 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T10:58:20.514 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T10:58:20.514 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T10:58:20.514 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T10:58:20.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T10:58:20.555 INFO:teuthology.orchestra.run.smithi087.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T10:58:20.590 INFO:teuthology.orchestra.run.smithi087.stderr:done 2023-12-16T10:58:20.592 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2023-12-16T10:58:20.607 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.BRaaN3Pspt 2023-12-16T10:58:20.608 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BRaaN3Pspt 2023-12-16T10:58:20.676 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.BRaaN3Pspt 2023-12-16T10:58:20.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2023-12-16T10:58:20.820 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.BRaaN3Pspt is 6KB 2023-12-16T10:58:20.869 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.BRaaN3Pspt 2023-12-16T10:58:20.883 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T10:58:20.948 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T10:58:20.948 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T10:58:20.948 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T10:58:20.948 INFO:teuthology.orchestra.run.smithi087.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T10:58:20.949 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T10:58:21.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2023-12-16T10:58:50.709 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-16T10:58:50.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-16T10:58:50.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T10:58:51.096 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-16T10:58:51.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-16T10:58:51.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T10:59:25.333 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-12-16T10:59:34.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-16T10:59:34.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T10:59:51.122 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T11:00:00.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-16T11:00:00.130 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T11:00:00.469 DEBUG:teuthology.orchestra.run.smithi087:> true 2023-12-16T11:00:00.798 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-16T11:00:00.798 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-16T11:00:00.798 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2023-12-16T11:00:00.812 INFO:teuthology.orchestra.run.smithi087.stdout:5.14.0-391.el9.x86_64 2023-12-16T11:00:00.812 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T11:00:00.812 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T11:00:00.812 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T11:00:01.813 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-16T11:00:01.814 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T11:00:01.888 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2023-12-16T11:00:01.934 DEBUG:teuthology.parallel:result is None 2023-12-16T11:00:34.353 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T11:00:46.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-16T11:00:46.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T11:00:46.619 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-12-16T11:00:46.958 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-16T11:00:46.958 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-16T11:00:46.959 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-12-16T11:00:46.973 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-391.el9.x86_64 2023-12-16T11:00:46.974 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T11:00:46.974 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T11:00:46.974 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T11:00:47.975 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-16T11:00:47.975 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T11:00:48.081 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2023-12-16T11:00:48.112 DEBUG:teuthology.parallel:result is None 2023-12-16T11:00:48.112 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-16T11:00:48.119 INFO:teuthology.task.internal:Creating test directory... 2023-12-16T11:00:48.119 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T11:00:48.122 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T11:00:48.139 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-16T11:00:48.154 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-16T11:00:48.170 INFO:teuthology.task.internal:Creating archive directory... 2023-12-16T11:00:48.170 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T11:00:48.181 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T11:00:48.213 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-16T11:00:48.220 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-16T11:00:48.220 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T11:00:48.249 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T11:00:48.284 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T11:00:48.286 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T11:00:48.298 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T11:00:48.300 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T11:00:48.302 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-16T11:00:48.309 INFO:teuthology.task.internal:Configuring sudo... 2023-12-16T11:00:48.309 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T11:00:48.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T11:00:48.372 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-16T11:00:48.381 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-16T11:00:48.382 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T11:00:48.414 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T11:00:48.430 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T11:00:48.517 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T11:00:48.581 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T11:00:48.631 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T11:00:48.667 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:00:48.667 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T11:00:48.735 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T11:00:48.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T11:00:48.843 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T11:00:48.895 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T11:00:48.930 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:00:48.930 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T11:00:48.999 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2023-12-16T11:00:49.001 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2023-12-16T11:00:49.057 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T11:00:49.080 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T11:00:49.429 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-16T11:00:49.436 INFO:teuthology.task.internal:Starting timer... 2023-12-16T11:00:49.436 INFO:teuthology.run_tasks:Running task pcp... 2023-12-16T11:00:49.482 INFO:teuthology.run_tasks:Running task selinux... 2023-12-16T11:00:49.498 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2023-12-16T11:00:49.556 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2023-12-16T11:00:49.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2023-12-16T11:00:49.619 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2023-12-16T11:00:49.621 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-16T11:00:49.621 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2023-12-16T11:00:49.642 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2023-12-16T11:00:49.642 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2023-12-16T11:00:49.676 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2023-12-16T11:00:49.676 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2023-12-16T11:00:49.676 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T11:00:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T11:00:49.711 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T11:00:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T11:00:49.744 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-16T11:00:49.744 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2023-12-16T11:00:49.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2023-12-16T11:00:49.828 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-16T11:00:49.838 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-16T11:00:49.843 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-16T11:00:49.997 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-16T11:00:50.015 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-16T11:00:50.016 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-16T11:06:18.346 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2023-12-16T11:06:18.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-16T11:06:18.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T11:06:18.439 DEBUG:teuthology.orchestra.run.smithi087:> true 2023-12-16T11:06:18.503 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-12-16T11:06:18.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-16T11:06:18.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T11:06:18.637 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-12-16T11:06:18.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-12-16T11:06:18.709 INFO:teuthology.run_tasks:Running task clock... 2023-12-16T11:06:18.719 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-16T11:06:18.719 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T11:06:18.720 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T11:06:18.722 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T11:06:18.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T11:06:18.750 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T11:06:18.766 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T11:06:18.782 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T11:06:18.795 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2023-12-16T11:06:18.799 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T11:06:18.809 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2023-12-16T11:06:18.825 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T11:06:18.829 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2023-12-16T11:06:18.841 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T11:06:18.843 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2023-12-16T11:06:18.860 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T11:06:18.878 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T11:06:18.884 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2023-12-16T11:06:18.888 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T11:06:18.888 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2023-12-16T11:06:18.888 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.889 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.889 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.889 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.919 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2023-12-16T11:06:18.922 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T11:06:18.922 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2023-12-16T11:06:18.922 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.922 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.922 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.922 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:06:18.923 INFO:teuthology.run_tasks:Running task install... 2023-12-16T11:06:18.932 DEBUG:teuthology.task.install:project ceph 2023-12-16T11:06:18.933 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}} 2023-12-16T11:06:18.933 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'} 2023-12-16T11:06:18.933 INFO:teuthology.task.install:Using flavor: default 2023-12-16T11:06:18.938 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-16T11:06:18.938 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-16T11:06:18.939 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678', 'tag': None, 'wait_for_package': False} 2023-12-16T11:06:18.939 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T11:06:18.941 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678', 'tag': None, 'wait_for_package': False} 2023-12-16T11:06:18.941 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T11:06:19.112 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/ 2023-12-16T11:06:19.112 INFO:teuthology.task.install.rpm:Package version is 19.0.0-191.g6d933094 2023-12-16T11:06:19.181 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/ 2023-12-16T11:06:19.181 INFO:teuthology.task.install.rpm:Package version is 19.0.0-191.g6d933094 2023-12-16T11:06:19.232 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T11:06:19.232 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:06:19.232 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T11:06:19.260 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-16T11:06:19.260 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6d933094a156ec7587742e5aa25fba5ba27ec678/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T11:06:19.296 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/6d933094a156ec7587742e5aa25fba5ba27ec678/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T11:06:19.296 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:06:19.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T11:06:19.324 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-12-16T11:06:19.324 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6d933094a156ec7587742e5aa25fba5ba27ec678/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T11:06:19.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T11:06:19.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T11:06:19.408 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T11:06:19.471 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T11:06:19.475 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2023-12-16T11:06:19.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2023-12-16T11:06:19.538 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2023-12-16T11:06:19.540 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2023-12-16T11:06:19.717 INFO:teuthology.orchestra.run.smithi160.stdout:73 files removed 2023-12-16T11:06:19.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2023-12-16T11:06:19.788 INFO:teuthology.orchestra.run.smithi087.stdout:73 files removed 2023-12-16T11:06:19.815 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2023-12-16T11:06:20.401 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 233 kB/s | 79 kB 00:00 2023-12-16T11:06:20.597 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 168 kB/s | 79 kB 00:00 2023-12-16T11:06:20.751 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 31 kB/s | 9.5 kB 00:00 2023-12-16T11:06:21.023 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-12-16T11:06:21.118 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2023-12-16T11:06:21.357 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2023-12-16T11:06:21.521 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-12-16T11:06:21.767 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-16T11:06:22.821 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-9 - BaseOS 7.9 MB/s | 2.0 MB 00:00 2023-12-16T11:06:23.101 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-12-16T11:06:23.892 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 7.9 MB 00:00 2023-12-16T11:06:24.720 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 7.9 MB 00:01 2023-12-16T11:06:26.168 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - AppStream 15 MB/s | 18 MB 00:01 2023-12-16T11:06:30.011 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-12-16T11:06:30.529 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 18 MB 00:04 2023-12-16T11:06:32.536 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2023-12-16T11:06:33.691 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2023-12-16T11:06:34.422 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-12-16T11:06:35.988 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2023-12-16T11:06:38.338 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-12-16T11:06:39.392 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 6.0 MB/s | 20 MB 00:03 2023-12-16T11:06:40.793 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:06:40.799 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:06:40.799 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:06:40.799 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:06:40.799 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:06:40.799 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:19.0.0-191.g6d933094.el9 ceph 8.6 M 2023-12-16T11:06:40.799 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:19.0.0-191.g6d933094.el9 ceph 3.3 M 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:19.0.0-191.g6d933094.el9 ceph 3.0 M 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:19.0.0-191.g6d933094.el9 ceph 5.2 M 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:19.0.0-191.g6d933094.el9 ceph 19 M 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:19.0.0-191.g6d933094.el9 ceph 24 k 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:19.0.0-191.g6d933094.el9 ceph 719 k 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T11:06:40.800 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:19.0.0-191.g6d933094.el9 ceph 476 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:19.0.0-191.g6d933094.el9 ceph 4.8 M 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:19.0.0-191.g6d933094.el9 ceph 45 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:19.0.0-191.g6d933094.el9 ceph 130 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:19.0.0-191.g6d933094.el9 ceph 161 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:19.0.0-191.g6d933094.el9 ceph 321 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:19.0.0-191.g6d933094.el9 ceph 298 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:19.0.0-191.g6d933094.el9 ceph 99 k 2023-12-16T11:06:40.801 INFO:teuthology.orchestra.run.smithi087.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout:Install 27 Packages 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 55 M 2023-12-16T11:06:40.802 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:06:42.295 INFO:teuthology.orchestra.run.smithi087.stdout:(1/29): ceph-base-19.0.0-191.g6d933094.el9.x86_ 3.7 MB/s | 5.2 MB 00:01 2023-12-16T11:06:42.387 INFO:teuthology.orchestra.run.smithi087.stdout:(2/29): ceph-radosgw-19.0.0-191.g6d933094.el9.x 5.7 MB/s | 8.6 MB 00:01 2023-12-16T11:06:42.412 INFO:teuthology.orchestra.run.smithi087.stdout:(3/29): ceph-selinux-19.0.0-191.g6d933094.el9.x 205 kB/s | 24 kB 00:00 2023-12-16T11:06:42.679 INFO:teuthology.orchestra.run.smithi087.stdout:(4/29): ceph-common-19.0.0-191.g6d933094.el9.x8 11 MB/s | 19 MB 00:01 2023-12-16T11:06:42.704 INFO:teuthology.orchestra.run.smithi087.stdout:(5/29): libcephfs2-19.0.0-191.g6d933094.el9.x86 2.2 MB/s | 719 kB 00:00 2023-12-16T11:06:42.730 INFO:teuthology.orchestra.run.smithi087.stdout:(6/29): libradosstriper1-19.0.0-191.g6d933094.e 1.5 MB/s | 476 kB 00:00 2023-12-16T11:06:42.780 INFO:teuthology.orchestra.run.smithi087.stdout:(7/29): python3-ceph-argparse-19.0.0-191.g6d933 601 kB/s | 45 kB 00:00 2023-12-16T11:06:42.805 INFO:teuthology.orchestra.run.smithi087.stdout:(8/29): python3-ceph-common-19.0.0-191.g6d93309 1.7 MB/s | 130 kB 00:00 2023-12-16T11:06:42.855 INFO:teuthology.orchestra.run.smithi087.stdout:(9/29): python3-cephfs-19.0.0-191.g6d933094.el9 2.1 MB/s | 161 kB 00:00 2023-12-16T11:06:42.964 INFO:teuthology.orchestra.run.smithi087.stdout:(10/29): librgw2-19.0.0-191.g6d933094.el9.x86_6 17 MB/s | 4.8 MB 00:00 2023-12-16T11:06:42.990 INFO:teuthology.orchestra.run.smithi087.stdout:(11/29): python3-rados-19.0.0-191.g6d933094.el9 1.7 MB/s | 321 kB 00:00 2023-12-16T11:06:43.015 INFO:teuthology.orchestra.run.smithi087.stdout:(12/29): python3-rgw-19.0.0-191.g6d933094.el9.x 1.9 MB/s | 99 kB 00:00 2023-12-16T11:06:43.040 INFO:teuthology.orchestra.run.smithi087.stdout:(13/29): python3-rbd-19.0.0-191.g6d933094.el9.x 1.6 MB/s | 298 kB 00:00 2023-12-16T11:06:43.165 INFO:teuthology.orchestra.run.smithi087.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2023-12-16T11:06:43.190 INFO:teuthology.orchestra.run.smithi087.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 908 kB/s | 159 kB 00:00 2023-12-16T11:06:43.207 INFO:teuthology.orchestra.run.smithi087.stdout:(16/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 376 kB/s | 44 kB 00:00 2023-12-16T11:06:43.275 INFO:teuthology.orchestra.run.smithi087.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 661 kB 00:00 2023-12-16T11:06:43.308 INFO:teuthology.orchestra.run.smithi087.stdout:(18/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2023-12-16T11:06:43.325 INFO:teuthology.orchestra.run.smithi087.stdout:(19/29): python3-prettytable-0.7.2-27.el9.noarc 352 kB/s | 41 kB 00:00 2023-12-16T11:06:43.484 INFO:teuthology.orchestra.run.smithi087.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2023-12-16T11:06:43.601 INFO:teuthology.orchestra.run.smithi087.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2023-12-16T11:06:43.626 INFO:teuthology.orchestra.run.smithi087.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 881 kB/s | 309 kB 00:00 2023-12-16T11:06:43.693 INFO:teuthology.orchestra.run.smithi087.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2023-12-16T11:06:43.801 INFO:teuthology.orchestra.run.smithi087.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2023-12-16T11:06:43.826 INFO:teuthology.orchestra.run.smithi087.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2023-12-16T11:06:43.852 INFO:teuthology.orchestra.run.smithi087.stdout:(26/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2023-12-16T11:06:43.894 INFO:teuthology.orchestra.run.smithi087.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2023-12-16T11:06:44.077 INFO:teuthology.orchestra.run.smithi087.stdout:(28/29): librbd1-19.0.0-191.g6d933094.el9.x86_6 13 MB/s | 3.0 MB 00:00 2023-12-16T11:06:44.097 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2023-12-16T11:06:44.294 INFO:teuthology.orchestra.run.smithi087.stdout:(29/29): librados2-19.0.0-191.g6d933094.el9.x86 7.1 MB/s | 3.3 MB 00:00 2023-12-16T11:06:44.298 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:06:44.298 INFO:teuthology.orchestra.run.smithi087.stdout:Total 16 MB/s | 55 MB 00:03 2023-12-16T11:06:44.298 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:06:44.332 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:06:44.332 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:06:44.785 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:06:44.786 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:06:46.139 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:06:46.231 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-16T11:06:46.266 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-16T11:06:46.316 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-16T11:06:46.563 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:06:46.569 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:19.0.0-191.g6d933094.el9 ceph 8.6 M 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:19.0.0-191.g6d933094.el9 ceph 3.3 M 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:19.0.0-191.g6d933094.el9 ceph 3.0 M 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:19.0.0-191.g6d933094.el9 ceph 5.2 M 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:19.0.0-191.g6d933094.el9 ceph 19 M 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:19.0.0-191.g6d933094.el9 ceph 24 k 2023-12-16T11:06:46.570 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:19.0.0-191.g6d933094.el9 ceph 719 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:19.0.0-191.g6d933094.el9 ceph 476 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:19.0.0-191.g6d933094.el9 ceph 4.8 M 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:19.0.0-191.g6d933094.el9 ceph 45 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:19.0.0-191.g6d933094.el9 ceph 130 k 2023-12-16T11:06:46.571 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:19.0.0-191.g6d933094.el9 ceph 161 k 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:19.0.0-191.g6d933094.el9 ceph 321 k 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:19.0.0-191.g6d933094.el9 ceph 298 k 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:19.0.0-191.g6d933094.el9 ceph 99 k 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout:Install 27 Packages 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2023-12-16T11:06:46.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:06:46.573 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 55 M 2023-12-16T11:06:46.573 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:06:46.683 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-16T11:06:46.723 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:19.0.0-191.g6d933094.el9.x86_64 5/31 2023-12-16T11:06:46.807 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:19.0.0-191.g6d933094.el9.x86_64 5/31 2023-12-16T11:06:46.891 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 6/31 2023-12-16T11:06:46.925 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 7/31 2023-12-16T11:06:46.979 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 7/31 2023-12-16T11:06:46.999 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-16T11:06:47.038 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:19.0.0-191.g6d933094.el9.x 9/31 2023-12-16T11:06:47.104 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 10/31 2023-12-16T11:06:47.130 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 11/31 2023-12-16T11:06:47.196 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 11/31 2023-12-16T11:06:47.306 INFO:teuthology.orchestra.run.smithi160.stdout:(1/29): ceph-base-19.0.0-191.g6d933094.el9.x86_ 7.7 MB/s | 5.2 MB 00:00 2023-12-16T11:06:47.306 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-16T11:06:47.346 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-16T11:06:47.356 INFO:teuthology.orchestra.run.smithi160.stdout:(2/29): ceph-selinux-19.0.0-191.g6d933094.el9.x 482 kB/s | 24 kB 00:00 2023-12-16T11:06:47.381 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-16T11:06:47.411 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-16T11:06:47.439 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-16T11:06:47.465 INFO:teuthology.orchestra.run.smithi160.stdout:(3/29): libcephfs2-19.0.0-191.g6d933094.el9.x86 6.5 MB/s | 719 kB 00:00 2023-12-16T11:06:47.472 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-16T11:06:47.540 INFO:teuthology.orchestra.run.smithi160.stdout:(4/29): libradosstriper1-19.0.0-191.g6d933094.e 6.2 MB/s | 476 kB 00:00 2023-12-16T11:06:47.775 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-16T11:06:47.802 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:19.0.0-191.g6d933094.el9.x86_64 19/31 2023-12-16T11:06:47.884 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:19.0.0-191.g6d933094.el9.x86_64 19/31 2023-12-16T11:06:47.915 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 20/31 2023-12-16T11:06:47.951 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-16T11:06:48.073 INFO:teuthology.orchestra.run.smithi160.stdout:(5/29): librgw2-19.0.0-191.g6d933094.el9.x86_64 8.9 MB/s | 4.8 MB 00:00 2023-12-16T11:06:48.257 INFO:teuthology.orchestra.run.smithi160.stdout:(6/29): ceph-common-19.0.0-191.g6d933094.el9.x8 12 MB/s | 19 MB 00:01 2023-12-16T11:06:48.291 INFO:teuthology.orchestra.run.smithi160.stdout:(7/29): python3-ceph-argparse-19.0.0-191.g6d933 208 kB/s | 45 kB 00:00 2023-12-16T11:06:48.316 INFO:teuthology.orchestra.run.smithi160.stdout:(8/29): python3-ceph-common-19.0.0-191.g6d93309 2.2 MB/s | 130 kB 00:00 2023-12-16T11:06:48.341 INFO:teuthology.orchestra.run.smithi160.stdout:(9/29): python3-cephfs-19.0.0-191.g6d933094.el9 3.2 MB/s | 161 kB 00:00 2023-12-16T11:06:48.379 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-16T11:06:48.391 INFO:teuthology.orchestra.run.smithi160.stdout:(10/29): python3-rados-19.0.0-191.g6d933094.el9 4.2 MB/s | 321 kB 00:00 2023-12-16T11:06:48.404 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:19.0.0-191.g6d933094.el9.x86_64 23/31 2023-12-16T11:06:48.417 INFO:teuthology.orchestra.run.smithi160.stdout:(11/29): python3-rbd-19.0.0-191.g6d933094.el9.x 3.9 MB/s | 298 kB 00:00 2023-12-16T11:06:48.434 INFO:teuthology.orchestra.run.smithi160.stdout:(12/29): python3-rgw-19.0.0-191.g6d933094.el9.x 2.3 MB/s | 99 kB 00:00 2023-12-16T11:06:48.475 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:19.0.0-191.g6d933094.el9.x86_64 23/31 2023-12-16T11:06:48.542 INFO:teuthology.orchestra.run.smithi160.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 847 kB/s | 106 kB 00:00 2023-12-16T11:06:48.543 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 24/31 2023-12-16T11:06:48.567 INFO:teuthology.orchestra.run.smithi160.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2023-12-16T11:06:48.573 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:19.0.0-191.g6d933094.el9.x86 25/31 2023-12-16T11:06:48.611 INFO:teuthology.orchestra.run.smithi160.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 646 kB/s | 44 kB 00:00 2023-12-16T11:06:48.668 INFO:teuthology.orchestra.run.smithi160.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2023-12-16T11:06:48.693 INFO:teuthology.orchestra.run.smithi160.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 282 kB 00:00 2023-12-16T11:06:48.718 INFO:teuthology.orchestra.run.smithi160.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2023-12-16T11:06:49.335 INFO:teuthology.orchestra.run.smithi160.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 481 kB/s | 309 kB 00:00 2023-12-16T11:06:49.502 INFO:teuthology.orchestra.run.smithi160.stdout:(20/29): ceph-radosgw-19.0.0-191.g6d933094.el9. 3.0 MB/s | 8.6 MB 00:02 2023-12-16T11:06:49.519 INFO:teuthology.orchestra.run.smithi160.stdout:(21/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 141 kB/s | 26 kB 00:00 2023-12-16T11:06:49.636 INFO:teuthology.orchestra.run.smithi160.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2023-12-16T11:06:49.911 INFO:teuthology.orchestra.run.smithi160.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2023-12-16T11:06:49.928 INFO:teuthology.orchestra.run.smithi160.stdout:(24/29): liboath-2.6.7-2.el9.x86_64.rpm 114 kB/s | 49 kB 00:00 2023-12-16T11:06:49.995 INFO:teuthology.orchestra.run.smithi160.stdout:(25/29): libarrow-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2023-12-16T11:06:50.020 INFO:teuthology.orchestra.run.smithi160.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-12-16T11:06:50.313 INFO:teuthology.orchestra.run.smithi160.stdout:(27/29): librados2-19.0.0-191.g6d933094.el9.x86 10 MB/s | 3.3 MB 00:00 2023-12-16T11:06:50.438 INFO:teuthology.orchestra.run.smithi160.stdout:(28/29): librbd1-19.0.0-191.g6d933094.el9.x86_6 7.2 MB/s | 3.0 MB 00:00 2023-12-16T11:06:50.479 INFO:teuthology.orchestra.run.smithi160.stdout:(29/29): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2023-12-16T11:06:50.485 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:06:50.486 INFO:teuthology.orchestra.run.smithi160.stdout:Total 14 MB/s | 55 MB 00:03 2023-12-16T11:06:50.486 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:06:50.517 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:06:50.517 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:06:50.697 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 26/31 2023-12-16T11:06:50.728 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 26/31 2023-12-16T11:06:50.973 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:06:50.974 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:06:51.275 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 26/31 2023-12-16T11:06:51.314 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 27/31 2023-12-16T11:06:51.387 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 27/31 2023-12-16T11:06:51.388 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T11:06:51.388 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T11:06:51.388 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:06:51.424 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 28/31 2023-12-16T11:06:52.346 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:06:52.430 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-16T11:06:52.467 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-16T11:06:52.518 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-16T11:06:52.894 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-16T11:06:52.933 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:19.0.0-191.g6d933094.el9.x86_64 5/31 2023-12-16T11:06:53.032 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:19.0.0-191.g6d933094.el9.x86_64 5/31 2023-12-16T11:06:53.125 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 6/31 2023-12-16T11:06:53.167 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 7/31 2023-12-16T11:06:53.254 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 7/31 2023-12-16T11:06:53.283 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-16T11:06:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:19.0.0-191.g6d933094.el9.x 9/31 2023-12-16T11:06:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 10/31 2023-12-16T11:06:53.419 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 11/31 2023-12-16T11:06:53.484 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 11/31 2023-12-16T11:06:53.591 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-16T11:06:53.618 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-16T11:06:53.667 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-16T11:06:53.707 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-16T11:06:53.737 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-16T11:06:53.769 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-16T11:06:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-16T11:06:54.107 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:19.0.0-191.g6d933094.el9.x86_64 19/31 2023-12-16T11:06:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:19.0.0-191.g6d933094.el9.x86_64 19/31 2023-12-16T11:06:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 20/31 2023-12-16T11:06:54.272 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-16T11:06:54.692 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-16T11:06:54.726 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:19.0.0-191.g6d933094.el9.x86_64 23/31 2023-12-16T11:06:54.797 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:19.0.0-191.g6d933094.el9.x86_64 23/31 2023-12-16T11:06:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 24/31 2023-12-16T11:06:54.895 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:19.0.0-191.g6d933094.el9.x86 25/31 2023-12-16T11:06:57.119 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 26/31 2023-12-16T11:06:57.166 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 26/31 2023-12-16T11:06:57.714 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 26/31 2023-12-16T11:06:57.763 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 27/31 2023-12-16T11:06:57.992 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 27/31 2023-12-16T11:06:57.992 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T11:06:57.992 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T11:06:57.993 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:06:58.022 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 28/31 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 28/31 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /root 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2023-12-16T11:07:00.941 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:01.790 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 29/31 2023-12-16T11:07:01.814 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 29/31 2023-12-16T11:07:01.814 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:01.814 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T11:07:01.814 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T11:07:01.814 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T11:07:01.815 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:01.816 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-16T11:07:01.900 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-16T11:07:01.900 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 1/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 2/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 3/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 4/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 5/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 6/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:19.0.0-191.g6d933094.el9.x86_64 7/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:19.0.0-191.g6d933094.el9.x 8/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:19.0.0-191.g6d933094.el9.x86 9/31 2023-12-16T11:07:02.987 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 10/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 11/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 12/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 13/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-16T11:07:02.988 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-16T11:07:02.989 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-16T11:07:02.989 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:19.0.0-191.g6d933094.el9.x86_64 28/31 2023-12-16T11:07:02.989 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-16T11:07:02.989 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:19.0.0-191.g6d933094.el9.x86_64 30/31 2023-12-16T11:07:03.863 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-16T11:07:03.863 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:03.863 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2023-12-16T11:07:03.863 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.863 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.863 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.864 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: re2-1:20211101-3.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:03.865 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:04.167 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2023-12-16T11:07:04.719 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:26 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:04.863 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:04.864 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:04.864 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:04.864 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:04.864 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:04.864 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:19.0.0-191.g6d933094.el9 ceph 41 M 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:19.0.0-191.g6d933094.el9 ceph 163 k 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2023-12-16T11:07:04.865 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:04.866 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 42 M 2023-12-16T11:07:04.866 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 173 M 2023-12-16T11:07:04.866 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:05.011 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 186 kB 00:00 2023-12-16T11:07:05.053 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2023-12-16T11:07:05.086 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 8.8 MB/s | 300 kB 00:00 2023-12-16T11:07:05.120 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-16T11:07:05.271 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): libcephsqlite-19.0.0-191.g6d933094.el9.x 404 kB/s | 163 kB 00:00 2023-12-16T11:07:07.137 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-test-19.0.0-191.g6d933094.el9.x86_6 18 MB/s | 41 MB 00:02 2023-12-16T11:07:07.140 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:07.141 INFO:teuthology.orchestra.run.smithi087.stdout:Total 18 MB/s | 42 MB 00:02 2023-12-16T11:07:07.141 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:07.160 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:07.160 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:07.520 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:07.521 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 28/31 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /root 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2023-12-16T11:07:07.682 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:08.153 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-16T11:07:08.196 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-16T11:07:08.228 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-16T11:07:08.271 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-16T11:07:08.298 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 5/6 2023-12-16T11:07:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 29/31 2023-12-16T11:07:08.545 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 29/31 2023-12-16T11:07:08.545 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:08.545 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T11:07:08.545 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T11:07:08.545 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T11:07:08.545 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:08.547 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-16T11:07:08.632 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-16T11:07:08.632 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-16T11:07:09.771 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-16T11:07:09.771 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 1/31 2023-12-16T11:07:09.771 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 2/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 3/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 4/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 5/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 6/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:19.0.0-191.g6d933094.el9.x86_64 7/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:19.0.0-191.g6d933094.el9.x 8/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:19.0.0-191.g6d933094.el9.x86 9/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 10/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 11/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 12/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 13/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-16T11:07:09.772 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:19.0.0-191.g6d933094.el9.x86_64 28/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-16T11:07:09.773 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:19.0.0-191.g6d933094.el9.x86_64 30/31 2023-12-16T11:07:10.822 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-16T11:07:10.822 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:10.822 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2023-12-16T11:07:10.822 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.822 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.822 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:07:10.822 INFO:teuthology.orchestra.run.smithi160.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T11:07:10.823 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: re2-1:20211101-3.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:10.824 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:11.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2023-12-16T11:07:11.742 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:27 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:11.886 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:19.0.0-191.g6d933094.el9 ceph 41 M 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:19.0.0-191.g6d933094.el9 ceph 163 k 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T11:07:11.887 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 42 M 2023-12-16T11:07:11.888 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 173 M 2023-12-16T11:07:11.889 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:07:12.053 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-16T11:07:12.094 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2023-12-16T11:07:12.136 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-16T11:07:12.170 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-16T11:07:12.249 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 5/6 2023-12-16T11:07:12.303 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): libcephsqlite-19.0.0-191.g6d933094.el9.x 395 kB/s | 163 kB 00:00 2023-12-16T11:07:12.322 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 6/6 2023-12-16T11:07:14.204 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): ceph-test-19.0.0-191.g6d933094.el9.x86_6 18 MB/s | 41 MB 00:02 2023-12-16T11:07:14.207 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:14.208 INFO:teuthology.orchestra.run.smithi160.stdout:Total 18 MB/s | 42 MB 00:02 2023-12-16T11:07:14.208 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:07:14.228 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:07:14.228 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:07:14.384 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 6/6 2023-12-16T11:07:14.384 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 1/6 2023-12-16T11:07:14.384 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 2/6 2023-12-16T11:07:14.384 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-16T11:07:14.384 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-16T11:07:14.384 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-16T11:07:14.593 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:07:14.594 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:07:14.755 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-16T11:07:14.755 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:14.755 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:14.755 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:14.755 INFO:teuthology.orchestra.run.smithi087.stdout: jq-1.6-16.el9.x86_64 2023-12-16T11:07:14.755 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:14.756 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T11:07:14.756 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T11:07:14.756 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T11:07:14.756 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:14.756 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:15.094 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2023-12-16T11:07:15.136 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:07:15.213 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-16T11:07:15.246 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-16T11:07:15.278 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-16T11:07:15.320 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-16T11:07:15.348 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 5/6 2023-12-16T11:07:15.646 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:37 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:15.792 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:19.0.0-191.g6d933094.el9 ceph 6.5 k 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:19.0.0-191.g6d933094.el9 ceph 2.1 M 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:19.0.0-191.g6d933094.el9 ceph 1.5 M 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 243 k 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:19.0.0-191.g6d933094.el9 ceph 4.4 M 2023-12-16T11:07:15.795 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:19.0.0-191.g6d933094.el9 ceph 18 M 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T11:07:15.796 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T11:07:15.797 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================== 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout:Install 35 Packages 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 30 M 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 111 M 2023-12-16T11:07:15.798 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:16.039 INFO:teuthology.orchestra.run.smithi087.stdout:(1/35): ceph-19.0.0-191.g6d933094.el9.x86_64.rp 41 kB/s | 6.5 kB 00:00 2023-12-16T11:07:16.272 INFO:teuthology.orchestra.run.smithi087.stdout:(2/35): ceph-mds-19.0.0-191.g6d933094.el9.x86_6 5.3 MB/s | 2.1 MB 00:00 2023-12-16T11:07:16.656 INFO:teuthology.orchestra.run.smithi087.stdout:(3/35): ceph-mon-19.0.0-191.g6d933094.el9.x86_6 7.1 MB/s | 4.4 MB 00:00 2023-12-16T11:07:16.756 INFO:teuthology.orchestra.run.smithi087.stdout:(4/35): ceph-mgr-modules-core-19.0.0-191.g6d933 2.4 MB/s | 243 kB 00:00 2023-12-16T11:07:16.831 INFO:teuthology.orchestra.run.smithi087.stdout:(5/35): ceph-mgr-19.0.0-191.g6d933094.el9.x86_6 1.5 MB/s | 1.5 MB 00:00 2023-12-16T11:07:16.890 INFO:teuthology.orchestra.run.smithi087.stdout:(6/35): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2023-12-16T11:07:16.913 INFO:teuthology.orchestra.run.smithi087.stdout:(7/35): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-12-16T11:07:16.938 INFO:teuthology.orchestra.run.smithi087.stdout:(8/35): python3-markupsafe-1.1.1-12.el9.x86_64. 660 kB/s | 32 kB 00:00 2023-12-16T11:07:16.955 INFO:teuthology.orchestra.run.smithi087.stdout:(9/35): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2023-12-16T11:07:16.972 INFO:teuthology.orchestra.run.smithi087.stdout:(10/35): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-16T11:07:16.997 INFO:teuthology.orchestra.run.smithi087.stdout:(11/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-12-16T11:07:17.039 INFO:teuthology.orchestra.run.smithi087.stdout:(12/35): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-16T11:07:17.064 INFO:teuthology.orchestra.run.smithi087.stdout:(13/35): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2023-12-16T11:07:17.090 INFO:teuthology.orchestra.run.smithi087.stdout:(14/35): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-16T11:07:17.115 INFO:teuthology.orchestra.run.smithi087.stdout:(15/35): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-16T11:07:17.140 INFO:teuthology.orchestra.run.smithi087.stdout:(16/35): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-16T11:07:17.332 INFO:teuthology.orchestra.run.smithi087.stdout:(17/35): ceph-osd-19.0.0-191.g6d933094.el9.x86_ 17 MB/s | 18 MB 00:01 2023-12-16T11:07:17.582 INFO:teuthology.orchestra.run.smithi087.stdout:(18/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 98 kB/s | 43 kB 00:00 2023-12-16T11:07:17.816 INFO:teuthology.orchestra.run.smithi087.stdout:(19/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 355 kB/s | 172 kB 00:00 2023-12-16T11:07:17.841 INFO:teuthology.orchestra.run.smithi087.stdout:(20/35): luarocks-3.9.2-1.el9.noarch.rpm 208 kB/s | 151 kB 00:00 2023-12-16T11:07:17.917 INFO:teuthology.orchestra.run.smithi087.stdout:(21/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 106 kB/s | 11 kB 00:00 2023-12-16T11:07:17.942 INFO:teuthology.orchestra.run.smithi087.stdout:(22/35): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2023-12-16T11:07:17.975 INFO:teuthology.orchestra.run.smithi087.stdout:(23/35): python3-cherrypy-18.6.1-2.el9.noarch.r 913 kB/s | 358 kB 00:00 2023-12-16T11:07:18.026 INFO:teuthology.orchestra.run.smithi087.stdout:(24/35): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2023-12-16T11:07:18.051 INFO:teuthology.orchestra.run.smithi087.stdout:(25/35): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2023-12-16T11:07:18.076 INFO:teuthology.orchestra.run.smithi087.stdout:(26/35): python3-jaraco-text-3.2.0-6.el9.noarch 195 kB/s | 20 kB 00:00 2023-12-16T11:07:18.143 INFO:teuthology.orchestra.run.smithi087.stdout:(27/35): python3-logutils-0.3.5-21.el9.noarch.r 396 kB/s | 46 kB 00:00 2023-12-16T11:07:18.168 INFO:teuthology.orchestra.run.smithi087.stdout:(28/35): python3-more-itertools-8.12.0-2.el9.no 674 kB/s | 79 kB 00:00 2023-12-16T11:07:18.219 INFO:teuthology.orchestra.run.smithi087.stdout:(29/35): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2023-12-16T11:07:18.244 INFO:teuthology.orchestra.run.smithi087.stdout:(30/35): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2023-12-16T11:07:18.294 INFO:teuthology.orchestra.run.smithi087.stdout:(31/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 717 kB/s | 90 kB 00:00 2023-12-16T11:07:18.319 INFO:teuthology.orchestra.run.smithi087.stdout:(32/35): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2023-12-16T11:07:18.430 INFO:teuthology.orchestra.run.smithi087.stdout:(33/35): python3-zc-lockfile-2.0-10.el9.noarch. 181 kB/s | 20 kB 00:00 2023-12-16T11:07:18.453 INFO:teuthology.orchestra.run.smithi087.stdout:(34/35): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2023-12-16T11:07:18.546 INFO:teuthology.orchestra.run.smithi087.stdout:(35/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2023-12-16T11:07:18.549 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:18.549 INFO:teuthology.orchestra.run.smithi087.stdout:Total 11 MB/s | 30 MB 00:02 2023-12-16T11:07:18.550 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:18.589 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:18.590 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:18.995 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:18.996 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:19.235 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 5/6 2023-12-16T11:07:19.314 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 6/6 2023-12-16T11:07:20.316 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:20.373 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-16T11:07:20.405 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-16T11:07:20.439 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-16T11:07:20.530 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-16T11:07:20.600 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-16T11:07:20.641 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-16T11:07:20.713 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-16T11:07:20.766 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-16T11:07:20.805 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-16T11:07:20.854 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-16T11:07:20.912 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-16T11:07:21.110 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-16T11:07:21.168 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-16T11:07:21.211 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-16T11:07:21.249 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-16T11:07:21.279 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-16T11:07:21.309 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 6/6 2023-12-16T11:07:21.309 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 1/6 2023-12-16T11:07:21.309 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 2/6 2023-12-16T11:07:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-16T11:07:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-16T11:07:21.310 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-16T11:07:21.354 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-16T11:07:21.389 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-16T11:07:21.414 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-16T11:07:21.455 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-16T11:07:21.489 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-16T11:07:21.524 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-16T11:07:21.614 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-16T11:07:21.655 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-16T11:07:21.726 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-16T11:07:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-16T11:07:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:21.732 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:07:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: jq-1.6-16.el9.x86_64 2023-12-16T11:07:21.732 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T11:07:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T11:07:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T11:07:21.733 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:21.733 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:21.865 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-16T11:07:21.987 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-16T11:07:22.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2023-12-16T11:07:22.146 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-191.g6d933094.el9.n 28/35 2023-12-16T11:07:22.176 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 29/35 2023-12-16T11:07:22.201 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 29/35 2023-12-16T11:07:22.201 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:22.201 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T11:07:22.201 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T11:07:22.201 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T11:07:22.201 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:22.283 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-16T11:07:22.575 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:38 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:22.726 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================== 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================== 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:19.0.0-191.g6d933094.el9 ceph 6.5 k 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:19.0.0-191.g6d933094.el9 ceph 2.1 M 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:19.0.0-191.g6d933094.el9 ceph 1.5 M 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 243 k 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:19.0.0-191.g6d933094.el9 ceph 4.4 M 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:19.0.0-191.g6d933094.el9 ceph 18 M 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-16T11:07:22.729 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T11:07:22.730 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T11:07:22.731 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T11:07:22.732 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T11:07:22.732 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:22.732 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:07:22.732 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================== 2023-12-16T11:07:22.732 INFO:teuthology.orchestra.run.smithi160.stdout:Install 35 Packages 2023-12-16T11:07:22.732 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:22.732 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 30 M 2023-12-16T11:07:22.733 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 111 M 2023-12-16T11:07:22.733 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:07:22.946 INFO:teuthology.orchestra.run.smithi160.stdout:(1/35): ceph-19.0.0-191.g6d933094.el9.x86_64.rp 41 kB/s | 6.5 kB 00:00 2023-12-16T11:07:23.180 INFO:teuthology.orchestra.run.smithi160.stdout:(2/35): ceph-mds-19.0.0-191.g6d933094.el9.x86_6 5.3 MB/s | 2.1 MB 00:00 2023-12-16T11:07:23.672 INFO:teuthology.orchestra.run.smithi160.stdout:(3/35): ceph-mgr-19.0.0-191.g6d933094.el9.x86_6 1.7 MB/s | 1.5 MB 00:00 2023-12-16T11:07:23.730 INFO:teuthology.orchestra.run.smithi160.stdout:(4/35): ceph-mon-19.0.0-191.g6d933094.el9.x86_6 5.6 MB/s | 4.4 MB 00:00 2023-12-16T11:07:23.814 INFO:teuthology.orchestra.run.smithi160.stdout:(5/35): ceph-mgr-modules-core-19.0.0-191.g6d933 1.7 MB/s | 243 kB 00:00 2023-12-16T11:07:23.850 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-16T11:07:23.864 INFO:teuthology.orchestra.run.smithi160.stdout:(6/35): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-12-16T11:07:23.879 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 32/35 2023-12-16T11:07:23.889 INFO:teuthology.orchestra.run.smithi160.stdout:(7/35): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2023-12-16T11:07:23.907 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 32/35 2023-12-16T11:07:23.907 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:23.907 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T11:07:23.907 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T11:07:23.907 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T11:07:23.907 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:23.914 INFO:teuthology.orchestra.run.smithi160.stdout:(8/35): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2023-12-16T11:07:23.940 INFO:teuthology.orchestra.run.smithi160.stdout:(9/35): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2023-12-16T11:07:23.965 INFO:teuthology.orchestra.run.smithi160.stdout:(10/35): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2023-12-16T11:07:23.990 INFO:teuthology.orchestra.run.smithi160.stdout:(11/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-16T11:07:24.183 INFO:teuthology.orchestra.run.smithi160.stdout:(12/35): ceph-osd-19.0.0-191.g6d933094.el9.x86_ 18 MB/s | 18 MB 00:01 2023-12-16T11:07:24.216 INFO:teuthology.orchestra.run.smithi160.stdout:(13/35): python3-cryptography-36.0.1-4.el9.x86_ 4.6 MB/s | 1.1 MB 00:00 2023-12-16T11:07:24.241 INFO:teuthology.orchestra.run.smithi160.stdout:(14/35): python3-ply-3.11-14.el9.noarch.rpm 412 kB/s | 103 kB 00:00 2023-12-16T11:07:24.266 INFO:teuthology.orchestra.run.smithi160.stdout:(15/35): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2023-12-16T11:07:24.291 INFO:teuthology.orchestra.run.smithi160.stdout:(16/35): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-16T11:07:24.317 INFO:teuthology.orchestra.run.smithi160.stdout:(17/35): python3-urllib3-1.26.5-4.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2023-12-16T11:07:24.347 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 33/35 2023-12-16T11:07:24.370 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 33/35 2023-12-16T11:07:24.370 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:24.370 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T11:07:24.370 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T11:07:24.370 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T11:07:24.370 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:24.572 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 34/35 2023-12-16T11:07:24.595 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 34/35 2023-12-16T11:07:24.595 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:24.595 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T11:07:24.595 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T11:07:24.596 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T11:07:24.596 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:24.661 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:19.0.0-191.g6d933094.el9.x86_64 35/35 2023-12-16T11:07:24.700 INFO:teuthology.orchestra.run.smithi160.stdout:(18/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 kB/s | 43 kB 00:00 2023-12-16T11:07:24.876 INFO:teuthology.orchestra.run.smithi160.stdout:(19/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 308 kB/s | 172 kB 00:00 2023-12-16T11:07:24.901 INFO:teuthology.orchestra.run.smithi160.stdout:(20/35): luarocks-3.9.2-1.el9.noarch.rpm 238 kB/s | 151 kB 00:00 2023-12-16T11:07:24.976 INFO:teuthology.orchestra.run.smithi160.stdout:(21/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2023-12-16T11:07:24.993 INFO:teuthology.orchestra.run.smithi160.stdout:(22/35): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2023-12-16T11:07:25.044 INFO:teuthology.orchestra.run.smithi160.stdout:(23/35): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2023-12-16T11:07:25.069 INFO:teuthology.orchestra.run.smithi160.stdout:(24/35): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2023-12-16T11:07:25.094 INFO:teuthology.orchestra.run.smithi160.stdout:(25/35): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2023-12-16T11:07:25.220 INFO:teuthology.orchestra.run.smithi160.stdout:(26/35): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2023-12-16T11:07:25.220 INFO:teuthology.orchestra.run.smithi160.stdout:(27/35): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2023-12-16T11:07:25.220 INFO:teuthology.orchestra.run.smithi160.stdout:(28/35): python3-more-itertools-8.12.0-2.el9.no 629 kB/s | 79 kB 00:00 2023-12-16T11:07:25.279 INFO:teuthology.orchestra.run.smithi160.stdout:(29/35): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2023-12-16T11:07:25.312 INFO:teuthology.orchestra.run.smithi160.stdout:(30/35): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2023-12-16T11:07:25.345 INFO:teuthology.orchestra.run.smithi160.stdout:(31/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2023-12-16T11:07:25.387 INFO:teuthology.orchestra.run.smithi160.stdout:(32/35): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2023-12-16T11:07:25.488 INFO:teuthology.orchestra.run.smithi160.stdout:(33/35): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2023-12-16T11:07:25.513 INFO:teuthology.orchestra.run.smithi160.stdout:(34/35): python3-zc-lockfile-2.0-10.el9.noarch. 159 kB/s | 20 kB 00:00 2023-12-16T11:07:25.621 INFO:teuthology.orchestra.run.smithi160.stdout:(35/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2023-12-16T11:07:25.628 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:25.628 INFO:teuthology.orchestra.run.smithi160.stdout:Total 10 MB/s | 30 MB 00:02 2023-12-16T11:07:25.629 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:07:25.668 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:07:25.668 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:07:26.080 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:07:26.081 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:07:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:19.0.0-191.g6d933094.el9.x86_64 35/35 2023-12-16T11:07:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:19.0.0-191.g6d933094.el9.x86_64 1/35 2023-12-16T11:07:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 2/35 2023-12-16T11:07:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 3/35 2023-12-16T11:07:26.484 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 4/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 5/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-191.g6d933094.el9.n 6/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-16T11:07:26.485 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-16T11:07:26.486 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-16T11:07:27.285 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: lua-5.4.4-4.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T11:07:27.286 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T11:07:27.287 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:27.288 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:27.364 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:07:27.424 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-16T11:07:27.465 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-16T11:07:27.514 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-16T11:07:27.605 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-16T11:07:27.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2023-12-16T11:07:27.684 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-16T11:07:27.725 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-16T11:07:27.796 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-16T11:07:27.859 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-16T11:07:27.898 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-16T11:07:27.946 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-16T11:07:28.002 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-16T11:07:28.172 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:50 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:28.190 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-16T11:07:28.235 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-16T11:07:28.265 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:07:28.268 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-16T11:07:28.300 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-16T11:07:28.314 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:28.315 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:07:28.315 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:28.330 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-16T11:07:28.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2023-12-16T11:07:28.398 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-16T11:07:28.432 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-16T11:07:28.465 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-16T11:07:28.506 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-16T11:07:28.541 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-16T11:07:28.575 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-16T11:07:28.664 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-16T11:07:28.698 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-16T11:07:28.769 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-16T11:07:28.906 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-16T11:07:28.931 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:50 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:29.031 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 808 k 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:07:29.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:29.077 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 808 k 2023-12-16T11:07:29.077 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 823 k 2023-12-16T11:07:29.077 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:29.184 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-191.g6d933094.el9.n 28/35 2023-12-16T11:07:29.211 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 29/35 2023-12-16T11:07:29.236 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 29/35 2023-12-16T11:07:29.236 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:29.236 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T11:07:29.236 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T11:07:29.236 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T11:07:29.236 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:29.318 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-16T11:07:29.354 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-19.0.0-191.g6d933094.el9.noarch.rpm 2.9 MB/s | 808 kB 00:00 2023-12-16T11:07:29.354 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:29.355 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.8 MB/s | 808 kB 00:00 2023-12-16T11:07:29.355 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:29.360 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:29.360 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:29.366 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:29.366 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:29.442 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:29.882 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:30.330 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:30.657 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:30.657 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:30.657 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:30.657 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:30.657 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:30.657 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:30.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2023-12-16T11:07:30.855 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-16T11:07:30.881 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 32/35 2023-12-16T11:07:30.906 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 32/35 2023-12-16T11:07:30.906 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:30.906 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T11:07:30.906 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T11:07:30.906 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T11:07:30.907 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:31.382 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 33/35 2023-12-16T11:07:31.395 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:53 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:31.405 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 33/35 2023-12-16T11:07:31.405 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:31.406 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T11:07:31.406 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T11:07:31.406 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T11:07:31.406 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:31.541 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:31.541 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-191.g6d933094.el9 ceph 141 k 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:07:31.542 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:31.543 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 141 k 2023-12-16T11:07:31.543 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 395 k 2023-12-16T11:07:31.543 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 34/35 2023-12-16T11:07:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 34/35 2023-12-16T11:07:31.639 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:31.639 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T11:07:31.639 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T11:07:31.639 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T11:07:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:31.707 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:19.0.0-191.g6d933094.el9.x86_64 35/35 2023-12-16T11:07:31.767 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-19.0.0-191.g6d93309 628 kB/s | 141 kB 00:00 2023-12-16T11:07:31.768 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:31.768 INFO:teuthology.orchestra.run.smithi087.stdout:Total 624 kB/s | 141 kB 00:00 2023-12-16T11:07:31.768 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:31.775 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:31.776 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:31.827 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:31.827 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:31.965 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:32.035 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-191.g6d933094.e 1/1 2023-12-16T11:07:32.058 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-191.g6d933094.e 1/1 2023-12-16T11:07:32.058 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:32.058 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T11:07:32.059 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:33.039 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-191.g6d933094.e 1/1 2023-12-16T11:07:33.039 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:33.039 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:33.039 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:33.039 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:33.039 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:33.260 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:19.0.0-191.g6d933094.el9.x86_64 35/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:19.0.0-191.g6d933094.el9.x86_64 1/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 2/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 3/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 4/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 5/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-191.g6d933094.el9.n 6/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-16T11:07:33.374 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-16T11:07:33.375 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-16T11:07:33.376 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-16T11:07:33.829 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:55 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:33.928 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:07:33.977 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:33.978 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:07:33.978 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:34.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2023-12-16T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-16T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:34.304 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: lua-5.4.4-4.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T11:07:34.305 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T11:07:34.306 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:34.307 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:34.577 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:56 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:34.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2023-12-16T11:07:34.724 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:34.725 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:34.725 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:34.725 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:34.725 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:34.725 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 1.8 M 2023-12-16T11:07:34.725 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:07:34.725 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 28 k 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 15 k 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout:Install 7 Packages 2023-12-16T11:07:34.726 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:34.727 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.1 M 2023-12-16T11:07:34.727 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 8.7 M 2023-12-16T11:07:34.727 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:35.080 INFO:teuthology.orchestra.run.smithi087.stdout:(1/7): ceph-prometheus-alerts-19.0.0-191.g6d933 77 kB/s | 15 kB 00:00 2023-12-16T11:07:35.121 INFO:teuthology.orchestra.run.smithi087.stdout:(2/7): ceph-grafana-dashboards-19.0.0-191.g6d93 118 kB/s | 28 kB 00:00 2023-12-16T11:07:35.192 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:51 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:35.285 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:07:35.305 INFO:teuthology.orchestra.run.smithi087.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 40 kB/s | 9.0 kB 00:00 2023-12-16T11:07:35.335 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:35.336 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:07:35.336 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:35.397 INFO:teuthology.orchestra.run.smithi087.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 149 kB/s | 41 kB 00:00 2023-12-16T11:07:35.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2023-12-16T11:07:35.430 INFO:teuthology.orchestra.run.smithi087.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 246 kB/s | 31 kB 00:00 2023-12-16T11:07:35.588 INFO:teuthology.orchestra.run.smithi087.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 984 kB/s | 188 kB 00:00 2023-12-16T11:07:35.705 INFO:teuthology.orchestra.run.smithi087.stdout:(7/7): ceph-mgr-dashboard-19.0.0-191.g6d933094. 2.1 MB/s | 1.8 MB 00:00 2023-12-16T11:07:35.709 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:35.709 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.1 MB/s | 2.1 MB 00:00 2023-12-16T11:07:35.709 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:35.716 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:35.716 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:35.780 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:35.781 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:35.961 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:51 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:36.083 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:36.103 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 808 k 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:07:36.104 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:36.105 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:07:36.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:36.105 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 808 k 2023-12-16T11:07:36.105 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 823 k 2023-12-16T11:07:36.105 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:07:36.140 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-16T11:07:36.174 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-16T11:07:36.229 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-16T11:07:36.260 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-16T11:07:36.290 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-191.g6d933094.el9.no 5/7 2023-12-16T11:07:36.457 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-19.0.0-191.g6d933094.el9.noarch.rpm 2.2 MB/s | 808 kB 00:00 2023-12-16T11:07:36.458 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:36.458 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.2 MB/s | 808 kB 00:00 2023-12-16T11:07:36.458 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:07:36.463 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:07:36.463 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:07:36.469 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:07:36.470 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:07:36.527 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-191.g6d933094.el9.n 6/7 2023-12-16T11:07:36.548 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:07:36.559 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 7/7 2023-12-16T11:07:36.930 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:37.021 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:37.098 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 7/7 2023-12-16T11:07:37.098 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-191.g6d933094.el9.n 1/7 2023-12-16T11:07:37.098 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 2/7 2023-12-16T11:07:37.098 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-191.g6d933094.el9.no 3/7 2023-12-16T11:07:37.098 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-16T11:07:37.098 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-16T11:07:37.098 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-16T11:07:37.400 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T11:07:37.423 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:37.424 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:37.641 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-16T11:07:37.713 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:07:37.713 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:37.713 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:07:37.713 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:37.713 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:37.713 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:37.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2023-12-16T11:07:38.187 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:00 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:38.330 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:38.331 INFO:teuthology.orchestra.run.smithi087.stdout:=========================================================================================== 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout:=========================================================================================== 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 7.4 M 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T11:07:38.332 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T11:07:38.333 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T11:07:38.333 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T11:07:38.333 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:38.333 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:38.333 INFO:teuthology.orchestra.run.smithi087.stdout:=========================================================================================== 2023-12-16T11:07:38.333 INFO:teuthology.orchestra.run.smithi087.stdout:Install 12 Packages 2023-12-16T11:07:38.333 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:38.334 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 37 M 2023-12-16T11:07:38.334 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 209 M 2023-12-16T11:07:38.334 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:38.493 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:54 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:38.504 INFO:teuthology.orchestra.run.smithi087.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 249 kB/s | 32 kB 00:00 2023-12-16T11:07:38.537 INFO:teuthology.orchestra.run.smithi087.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2023-12-16T11:07:38.571 INFO:teuthology.orchestra.run.smithi087.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-16T11:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-191.g6d933094.el9 ceph 141 k 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 141 k 2023-12-16T11:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 395 k 2023-12-16T11:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:07:38.663 INFO:teuthology.orchestra.run.smithi087.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2023-12-16T11:07:38.738 INFO:teuthology.orchestra.run.smithi087.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2023-12-16T11:07:38.814 INFO:teuthology.orchestra.run.smithi087.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2023-12-16T11:07:38.839 INFO:teuthology.orchestra.run.smithi087.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-16T11:07:38.871 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-19.0.0-191.g6d93309 619 kB/s | 141 kB 00:00 2023-12-16T11:07:38.871 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:38.872 INFO:teuthology.orchestra.run.smithi160.stdout:Total 615 kB/s | 141 kB 00:00 2023-12-16T11:07:38.872 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:07:38.879 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:07:38.879 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:07:38.931 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:07:38.931 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:07:39.064 INFO:teuthology.orchestra.run.smithi087.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2023-12-16T11:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:07:39.098 INFO:teuthology.orchestra.run.smithi087.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2023-12-16T11:07:39.123 INFO:teuthology.orchestra.run.smithi087.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-12-16T11:07:39.130 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-191.g6d933094.e 1/1 2023-12-16T11:07:39.155 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-191.g6d933094.e 1/1 2023-12-16T11:07:39.155 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:07:39.155 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T11:07:39.155 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:39.265 INFO:teuthology.orchestra.run.smithi087.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2023-12-16T11:07:39.890 INFO:teuthology.orchestra.run.smithi087.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.9 MB/s | 7.4 MB 00:01 2023-12-16T11:07:39.893 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:39.893 INFO:teuthology.orchestra.run.smithi087.stdout:Total 24 MB/s | 37 MB 00:01 2023-12-16T11:07:39.906 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:39.931 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:39.931 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:40.204 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-191.g6d933094.e 1/1 2023-12-16T11:07:40.204 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:40.205 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:07:40.205 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:07:40.205 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:40.205 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:40.317 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:40.318 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:40.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2023-12-16T11:07:40.973 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:56 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:41.022 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:41.068 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:07:41.096 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-16T11:07:41.118 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:41.119 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:07:41.119 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:41.127 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-16T11:07:41.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2023-12-16T11:07:41.230 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-16T11:07:41.264 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-16T11:07:41.453 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T11:07:41.479 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T11:07:41.557 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-16T11:07:41.757 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:57 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:41.903 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 1.8 M 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 28 k 2023-12-16T11:07:41.904 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 15 k 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout:Installing weak dependencies: 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout:Install 7 Packages 2023-12-16T11:07:41.905 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:41.906 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 2.1 M 2023-12-16T11:07:41.906 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 8.7 M 2023-12-16T11:07:41.906 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:07:42.225 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-16T11:07:42.324 INFO:teuthology.orchestra.run.smithi160.stdout:(1/7): ceph-grafana-dashboards-19.0.0-191.g6d93 111 kB/s | 28 kB 00:00 2023-12-16T11:07:42.325 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-16T11:07:42.341 INFO:teuthology.orchestra.run.smithi160.stdout:(2/7): ceph-prometheus-alerts-19.0.0-191.g6d933 56 kB/s | 15 kB 00:00 2023-12-16T11:07:42.382 INFO:teuthology.orchestra.run.smithi160.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2023-12-16T11:07:42.416 INFO:teuthology.orchestra.run.smithi160.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 544 kB/s | 41 kB 00:00 2023-12-16T11:07:42.433 INFO:teuthology.orchestra.run.smithi160.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 611 kB/s | 31 kB 00:00 2023-12-16T11:07:42.492 INFO:teuthology.orchestra.run.smithi160.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2023-12-16T11:07:42.734 INFO:teuthology.orchestra.run.smithi160.stdout:(7/7): ceph-mgr-dashboard-19.0.0-191.g6d933094. 2.7 MB/s | 1.8 MB 00:00 2023-12-16T11:07:42.735 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:42.736 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.5 MB/s | 2.1 MB 00:00 2023-12-16T11:07:42.736 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:07:42.744 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:07:42.744 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:07:42.813 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:07:42.813 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:07:43.106 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:07:43.152 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-16T11:07:43.185 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-16T11:07:43.231 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-16T11:07:43.264 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-16T11:07:43.294 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-191.g6d933094.el9.no 5/7 2023-12-16T11:07:43.530 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-191.g6d933094.el9.n 6/7 2023-12-16T11:07:43.561 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-16T11:07:43.562 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 7/7 2023-12-16T11:07:44.146 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 7/7 2023-12-16T11:07:44.147 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-191.g6d933094.el9.n 1/7 2023-12-16T11:07:44.147 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 2/7 2023-12-16T11:07:44.147 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-191.g6d933094.el9.no 3/7 2023-12-16T11:07:44.147 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-16T11:07:44.147 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-16T11:07:44.147 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T11:07:44.463 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T11:07:44.464 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T11:07:44.464 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:44.464 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:44.596 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-16T11:07:44.622 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-191.g6d9330 12/12 2023-12-16T11:07:44.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-16T11:07:45.227 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:01 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:45.373 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:45.374 INFO:teuthology.orchestra.run.smithi160.stdout:=========================================================================================== 2023-12-16T11:07:45.374 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:07:45.374 INFO:teuthology.orchestra.run.smithi160.stdout:=========================================================================================== 2023-12-16T11:07:45.374 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 7.4 M 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T11:07:45.375 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout:=========================================================================================== 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout:Install 12 Packages 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 37 M 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 209 M 2023-12-16T11:07:45.376 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:07:45.557 INFO:teuthology.orchestra.run.smithi160.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2023-12-16T11:07:45.591 INFO:teuthology.orchestra.run.smithi160.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2023-12-16T11:07:45.658 INFO:teuthology.orchestra.run.smithi160.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-12-16T11:07:45.683 INFO:teuthology.orchestra.run.smithi160.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 380 kB/s | 34 kB 00:00 2023-12-16T11:07:45.800 INFO:teuthology.orchestra.run.smithi160.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 33 MB/s | 4.7 MB 00:00 2023-12-16T11:07:45.867 INFO:teuthology.orchestra.run.smithi160.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-12-16T11:07:45.892 INFO:teuthology.orchestra.run.smithi160.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-16T11:07:46.076 INFO:teuthology.orchestra.run.smithi160.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 75 MB/s | 16 MB 00:00 2023-12-16T11:07:46.109 INFO:teuthology.orchestra.run.smithi160.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2023-12-16T11:07:46.135 INFO:teuthology.orchestra.run.smithi160.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-12-16T11:07:46.777 INFO:teuthology.orchestra.run.smithi160.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 5.4 MB/s | 7.4 MB 00:01 2023-12-16T11:07:46.801 INFO:teuthology.orchestra.run.smithi160.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 355 kB/s | 245 kB 00:00 2023-12-16T11:07:46.805 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:46.805 INFO:teuthology.orchestra.run.smithi160.stdout:Total 26 MB/s | 37 MB 00:01 2023-12-16T11:07:46.818 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:07:46.843 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:07:46.843 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:07:47.030 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-191.g6d9330 12/12 2023-12-16T11:07:47.030 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-191.g6d9330 1/12 2023-12-16T11:07:47.030 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-16T11:07:47.030 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-16T11:07:47.030 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-16T11:07:47.030 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T11:07:47.030 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T11:07:47.031 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-16T11:07:47.031 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-16T11:07:47.031 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-16T11:07:47.031 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-16T11:07:47.031 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-16T11:07:47.227 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:07:47.227 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T11:07:47.478 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T11:07:47.479 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T11:07:47.479 INFO:teuthology.orchestra.run.smithi087.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T11:07:47.479 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T11:07:47.479 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T11:07:47.479 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T11:07:47.479 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:47.479 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:47.861 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2023-12-16T11:07:47.926 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:07:48.008 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-16T11:07:48.039 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-16T11:07:48.150 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-16T11:07:48.184 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-16T11:07:48.366 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T11:07:48.391 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T11:07:48.414 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:10 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:48.467 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-16T11:07:48.560 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:48.561 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2023-12-16T11:07:48.561 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:48.561 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2023-12-16T11:07:48.561 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:48.561 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 50 k 2023-12-16T11:07:48.561 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T11:07:48.562 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:48.563 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:48.563 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2023-12-16T11:07:48.563 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2023-12-16T11:07:48.563 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:48.563 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 2.0 M 2023-12-16T11:07:48.564 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 26 M 2023-12-16T11:07:48.564 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:49.029 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 219 kB/s | 24 kB 00:00 2023-12-16T11:07:49.054 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 132 kB/s | 18 kB 00:00 2023-12-16T11:07:49.105 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-12-16T11:07:49.138 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2023-12-16T11:07:49.164 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2023-12-16T11:07:49.188 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-16T11:07:49.189 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2023-12-16T11:07:49.239 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): ceph-mgr-rook-19.0.0-191.g6d933094.el9. 156 kB/s | 50 kB 00:00 2023-12-16T11:07:49.289 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-16T11:07:49.331 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 105 kB/s | 15 kB 00:00 2023-12-16T11:07:49.357 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2023-12-16T11:07:49.465 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 876 kB/s | 197 kB 00:00 2023-12-16T11:07:49.491 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 442 kB/s | 59 kB 00:00 2023-12-16T11:07:49.557 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 979 kB/s | 90 kB 00:00 2023-12-16T11:07:49.616 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2023-12-16T11:07:49.617 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:49.618 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2023-12-16T11:07:49.618 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:49.631 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:49.631 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:49.727 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:49.727 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:50.255 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:50.405 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-16T11:07:50.458 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-16T11:07:50.514 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-16T11:07:50.537 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-16T11:07:50.551 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-16T11:07:50.593 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-16T11:07:50.687 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-16T11:07:50.788 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-16T11:07:50.844 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-16T11:07:51.555 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-16T11:07:51.570 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-16T11:07:51.582 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-191.g6d9330 12/12 2023-12-16T11:07:51.615 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-16T11:07:51.649 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-16T11:07:51.697 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-16T11:07:51.718 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 13/13 2023-12-16T11:07:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 13/13 2023-12-16T11:07:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 1/13 2023-12-16T11:07:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-16T11:07:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-16T11:07:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-16T11:07:52.889 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-16T11:07:52.889 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:52.889 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:52.889 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:52.889 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:52.890 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:53.150 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2023-12-16T11:07:53.704 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:15 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:53.849 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:53.850 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 133 k 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:53.851 INFO:teuthology.orchestra.run.smithi087.stdout:======================================================================================= 2023-12-16T11:07:53.852 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2023-12-16T11:07:53.852 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:53.852 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 6.8 M 2023-12-16T11:07:53.852 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 33 M 2023-12-16T11:07:53.852 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-191.g6d9330 12/12 2023-12-16T11:07:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-191.g6d9330 1/12 2023-12-16T11:07:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-16T11:07:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-16T11:07:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-16T11:07:54.376 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-16T11:07:54.467 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-16T11:07:54.467 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:54.467 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:54.468 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:07:54.484 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-12-16T11:07:54.618 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): ceph-mgr-cephadm-19.0.0-191.g6d933094.el 332 kB/s | 133 kB 00:00 2023-12-16T11:07:54.781 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2023-12-16T11:07:55.340 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:11 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:07:55.488 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:07:55.489 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2023-12-16T11:07:55.489 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:07:55.489 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2023-12-16T11:07:55.489 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:07:55.489 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 50 k 2023-12-16T11:07:55.489 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:07:55.489 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2023-12-16T11:07:55.490 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2023-12-16T11:07:55.491 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:07:55.491 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 2.0 M 2023-12-16T11:07:55.491 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 26 M 2023-12-16T11:07:55.491 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:07:55.601 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 52 kB/s | 58 kB 00:01 2023-12-16T11:07:55.743 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 46 kB/s | 51 kB 00:01 2023-12-16T11:07:55.789 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2023-12-16T11:07:55.815 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2023-12-16T11:07:55.848 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2023-12-16T11:07:55.873 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2023-12-16T11:07:55.899 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-16T11:07:55.924 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 861 kB/s | 43 kB 00:00 2023-12-16T11:07:56.099 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): ceph-mgr-rook-19.0.0-191.g6d933094.el9. 114 kB/s | 50 kB 00:00 2023-12-16T11:07:56.183 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2023-12-16T11:07:56.208 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2023-12-16T11:07:56.260 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 291 kB/s | 548 kB 00:01 2023-12-16T11:07:56.265 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:56.265 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.8 MB/s | 6.8 MB 00:02 2023-12-16T11:07:56.265 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:07:56.275 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:07:56.276 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:07:56.369 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:07:56.370 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:07:56.408 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2023-12-16T11:07:56.584 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 408 kB/s | 197 kB 00:00 2023-12-16T11:07:56.600 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 467 kB/s | 90 kB 00:00 2023-12-16T11:07:56.674 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:07:56.794 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2023-12-16T11:07:56.796 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:56.797 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2023-12-16T11:07:56.797 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:07:56.811 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:07:56.811 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:07:56.842 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-16T11:07:56.886 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-16T11:07:56.912 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:07:56.913 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:07:57.408 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-16T11:07:57.437 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:07:57.488 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-16T11:07:57.550 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-16T11:07:57.573 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 6/6 2023-12-16T11:07:57.611 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-16T11:07:57.662 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-16T11:07:57.717 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-16T11:07:57.755 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-16T11:07:57.796 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-16T11:07:57.888 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-16T11:07:58.000 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-16T11:07:58.047 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-16T11:07:58.404 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 6/6 2023-12-16T11:07:58.405 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/6 2023-12-16T11:07:58.405 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-16T11:07:58.405 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-16T11:07:58.405 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-16T11:07:58.405 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T11:07:58.712 INFO:teuthology.orchestra.run.smithi087.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T11:07:58.713 INFO:teuthology.orchestra.run.smithi087.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T11:07:58.713 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:58.713 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:07:58.767 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-16T11:07:58.819 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-16T11:07:58.853 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-16T11:07:58.900 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-16T11:07:58.922 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 13/13 2023-12-16T11:07:58.959 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2023-12-16T11:07:59.509 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:21 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:07:59.653 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:07:59.654 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:59.654 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:07:59.654 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:59.654 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:07:59.654 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:19.0.0-191.g6d933094.el9 ceph 859 k 2023-12-16T11:07:59.654 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-12-16T11:07:59.654 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 938 k 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.7 M 2023-12-16T11:07:59.655 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:07:59.736 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 13/13 2023-12-16T11:07:59.736 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 1/13 2023-12-16T11:07:59.736 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-16T11:07:59.737 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-16T11:07:59.778 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 646 kB/s | 79 kB 00:00 2023-12-16T11:07:59.978 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-19.0.0-191.g6d933094.el9.x86_6 2.6 MB/s | 859 kB 00:00 2023-12-16T11:07:59.979 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:07:59.980 INFO:teuthology.orchestra.run.smithi087.stdout:Total 2.8 MB/s | 938 kB 00:00 2023-12-16T11:07:59.980 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:08:00.001 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:08:00.001 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:08:00.077 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:08:00.077 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T11:08:00.122 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:00.123 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:00.272 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:08:00.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2023-12-16T11:08:00.394 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-16T11:08:00.462 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 2/2 2023-12-16T11:08:00.953 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:16 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:01.104 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:01.104 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 133 k 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:01.105 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:01.106 INFO:teuthology.orchestra.run.smithi160.stdout:======================================================================================= 2023-12-16T11:08:01.106 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-12-16T11:08:01.106 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:01.106 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 6.8 M 2023-12-16T11:08:01.106 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 33 M 2023-12-16T11:08:01.106 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:01.195 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 2/2 2023-12-16T11:08:01.195 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/2 2023-12-16T11:08:01.438 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-16T11:08:01.512 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-16T11:08:01.512 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:01.512 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:08:01.512 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-16T11:08:01.512 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:01.512 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:01.563 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-12-16T11:08:01.680 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): ceph-mgr-cephadm-19.0.0-191.g6d933094.el 328 kB/s | 133 kB 00:00 2023-12-16T11:08:01.713 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 385 kB/s | 58 kB 00:00 2023-12-16T11:08:01.738 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2023-12-16T11:08:01.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-volume 2023-12-16T11:08:01.830 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2023-12-16T11:08:01.832 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:01.832 INFO:teuthology.orchestra.run.smithi160.stdout:Total 9.3 MB/s | 6.8 MB 00:00 2023-12-16T11:08:01.832 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:01.842 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:01.842 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:01.930 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:01.930 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:02.228 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:02.346 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:24 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:02.386 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-16T11:08:02.431 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-16T11:08:02.494 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 262 k 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 262 k 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 1.2 M 2023-12-16T11:08:02.495 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:08:02.749 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-volume-19.0.0-191.g6d933094.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2023-12-16T11:08:02.750 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:02.750 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-12-16T11:08:02.750 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:08:02.758 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:08:02.758 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:08:02.789 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:08:02.789 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:08:02.951 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-16T11:08:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:08:03.042 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-16T11:08:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:08:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:08:03.083 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:08:03.083 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-16T11:08:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:03.103 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-16T11:08:03.129 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 6/6 2023-12-16T11:08:03.988 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 6/6 2023-12-16T11:08:03.988 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 1/6 2023-12-16T11:08:03.988 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-16T11:08:03.988 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-16T11:08:03.988 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-16T11:08:03.988 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-16T11:08:04.087 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:08:04.087 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:04.088 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:08:04.088 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:08:04.088 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:04.088 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T11:08:04.309 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T11:08:04.310 INFO:teuthology.orchestra.run.smithi160.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T11:08:04.310 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:04.310 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:04.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2023-12-16T11:08:04.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2023-12-16T11:08:04.981 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:26 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:05.100 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:21 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:05.126 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:05.126 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:05.126 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-16T11:08:05.126 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:19.0.0-191.g6d933094.el9 ceph 126 k 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:05.127 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 126 k 2023-12-16T11:08:05.128 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 456 k 2023-12-16T11:08:05.128 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:08:05.245 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:19.0.0-191.g6d933094.el9 ceph 859 k 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2023-12-16T11:08:05.246 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:05.247 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 938 k 2023-12-16T11:08:05.247 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 2.7 M 2023-12-16T11:08:05.247 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:05.353 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-19.0.0-191.g6d933094.el9.x86_64. 563 kB/s | 126 kB 00:00 2023-12-16T11:08:05.353 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:05.354 INFO:teuthology.orchestra.run.smithi087.stdout:Total 559 kB/s | 126 kB 00:00 2023-12-16T11:08:05.354 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:08:05.359 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:08:05.359 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:08:05.365 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 667 kB/s | 79 kB 00:00 2023-12-16T11:08:05.414 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:08:05.414 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:08:05.557 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:08:05.565 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): ceph-fuse-19.0.0-191.g6d933094.el9.x86_6 2.6 MB/s | 859 kB 00:00 2023-12-16T11:08:05.566 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:05.566 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.9 MB/s | 938 kB 00:00 2023-12-16T11:08:05.567 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:05.586 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:05.587 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:05.661 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:05.661 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:05.676 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:05.861 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:05.997 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-16T11:08:06.065 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 2/2 2023-12-16T11:08:06.114 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:06.409 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:06.410 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:06.410 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:08:06.410 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:06.410 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:06.410 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:06.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2023-12-16T11:08:06.818 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 2/2 2023-12-16T11:08:06.818 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/2 2023-12-16T11:08:07.152 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-16T11:08:07.152 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:07.152 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:07.152 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:19.0.0-191.g6d933094.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-16T11:08:07.152 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:07.152 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:07.222 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:29 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:07.317 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:07.370 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:07.371 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:07.371 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:07.435 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2023-12-16T11:08:07.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-volume 2023-12-16T11:08:07.988 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:29 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:08.023 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:24 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:08.131 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:19.0.0-191.g6d933094.el9 ceph 31 k 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:08.132 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 31 k 2023-12-16T11:08:08.133 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 140 k 2023-12-16T11:08:08.133 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:08:08.168 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume noarch 2:19.0.0-191.g6d933094.el9 ceph-noarch 262 k 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:08:08.169 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:08.170 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 262 k 2023-12-16T11:08:08.170 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 1.2 M 2023-12-16T11:08:08.170 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:08.406 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-19.0.0-191.g6d933094.el9.x86_64 114 kB/s | 31 kB 00:00 2023-12-16T11:08:08.406 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:08.406 INFO:teuthology.orchestra.run.smithi087.stdout:Total 113 kB/s | 31 kB 00:00 2023-12-16T11:08:08.406 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:08:08.410 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:08:08.410 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:08:08.438 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:08:08.438 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:08:08.532 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:08:08.569 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-volume-19.0.0-191.g6d933094.el9.noarch.rpm 657 kB/s | 262 kB 00:00 2023-12-16T11:08:08.570 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:08.570 INFO:teuthology.orchestra.run.smithi160.stdout:Total 655 kB/s | 262 kB 00:00 2023-12-16T11:08:08.570 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:08.579 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:08.579 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:08.612 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:08.613 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:08.636 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:08.826 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:08:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:08:08.912 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:08:08.912 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-16T11:08:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:09.008 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:09.292 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:09.293 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:09.455 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2023-12-16T11:08:10.008 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 1/1 2023-12-16T11:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-volume-2:19.0.0-191.g6d933094.el9.noarch 2023-12-16T11:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:10.102 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:10.153 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:10.154 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:10.154 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:10.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2023-12-16T11:08:10.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2023-12-16T11:08:10.762 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:10.857 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:10.907 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:10.908 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:10.921 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:26 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:10.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2023-12-16T11:08:11.065 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:19.0.0-191.g6d933094.el9 ceph 126 k 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 126 k 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 456 k 2023-12-16T11:08:11.066 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:11.391 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-19.0.0-191.g6d933094.el9.x86_64. 391 kB/s | 126 kB 00:00 2023-12-16T11:08:11.391 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:11.392 INFO:teuthology.orchestra.run.smithi160.stdout:Total 388 kB/s | 126 kB 00:00 2023-12-16T11:08:11.392 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:11.397 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:11.397 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:11.452 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:11.452 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:11.530 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:11.604 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:11.624 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:11.675 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:11.676 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:11.676 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:11.697 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:11.743 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2023-12-16T11:08:12.128 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:12.295 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:34 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:12.390 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:12.439 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:12.440 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:12.441 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:12.445 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:12.445 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:12.445 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:12.446 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:12.446 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:12.446 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:12.502 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2023-12-16T11:08:12.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2023-12-16T11:08:13.055 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:35 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:13.147 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:13.197 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:13.198 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:13.198 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:13.265 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2023-12-16T11:08:13.268 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:29 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:13.364 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:13.414 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:13.415 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:13.415 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:13.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2023-12-16T11:08:13.815 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:35 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:13.909 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:13.959 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:13.960 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:13.960 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:14.023 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2023-12-16T11:08:14.043 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:30 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:14.186 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:14.186 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:19.0.0-191.g6d933094.el9 ceph 31 k 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 31 k 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 140 k 2023-12-16T11:08:14.187 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:14.360 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-19.0.0-191.g6d933094.el9.x86_64 181 kB/s | 31 kB 00:00 2023-12-16T11:08:14.360 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:14.361 INFO:teuthology.orchestra.run.smithi160.stdout:Total 179 kB/s | 31 kB 00:00 2023-12-16T11:08:14.361 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:14.364 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:14.395 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:14.395 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:14.490 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:14.576 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:36 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:14.721 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:19.0.0-191.g6d933094.el9 ceph 85 k 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:08:14.722 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2023-12-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 231 k 2023-12-16T11:08:14.723 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:08:14.933 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-19.0.0-191.g6d933094.el9.x86_64.rpm 404 kB/s | 85 kB 00:00 2023-12-16T11:08:14.934 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:14.934 INFO:teuthology.orchestra.run.smithi087.stdout:Total 402 kB/s | 85 kB 00:00 2023-12-16T11:08:14.934 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:08:14.941 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:08:14.941 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:08:14.995 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:08:14.996 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:08:15.046 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:08:15.223 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:15.315 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:15.315 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:15.315 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:15.315 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:15.315 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:15.315 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:15.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2023-12-16T11:08:15.606 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:15.911 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:08:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:15.912 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:16.044 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:16.138 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2023-12-16T11:08:16.139 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:16.189 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:16.190 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:16.190 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:16.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2023-12-16T11:08:16.706 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:38 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:16.805 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:16.858 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:16.858 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:16.858 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-12-16T11:08:16.858 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:16.858 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:08:16.858 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:19.0.0-191.g6d933094.el9 ceph 3.0 M 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.0 M 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 12 M 2023-12-16T11:08:16.859 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:08:16.900 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:16.950 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:16.951 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:16.951 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:17.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2023-12-16T11:08:17.411 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-19.0.0-191.g6d933094.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2023-12-16T11:08:17.411 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:17.412 INFO:teuthology.orchestra.run.smithi087.stdout:Total 5.4 MB/s | 3.0 MB 00:00 2023-12-16T11:08:17.412 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:08:17.421 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:08:17.421 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:08:17.493 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:08:17.493 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:08:17.571 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:17.664 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:17.715 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:17.716 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:17.780 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2023-12-16T11:08:17.916 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:08:17.970 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:17.994 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:17.994 INFO:teuthology.orchestra.run.smithi087.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:08:17.994 INFO:teuthology.orchestra.run.smithi087.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T11:08:17.994 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T11:08:17.994 INFO:teuthology.orchestra.run.smithi087.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T11:08:17.994 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:18.335 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:34 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:18.430 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:18.481 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:18.482 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:18.482 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:18.551 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2023-12-16T11:08:19.108 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:35 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:19.111 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:19.111 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:19.111 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:08:19.111 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:19.111 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:19.112 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:19.201 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:19.253 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:19.254 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:19.254 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:19.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2023-12-16T11:08:19.360 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2023-12-16T11:08:19.873 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:35 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:19.925 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:41 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:19.967 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:20.017 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:20.018 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:20.018 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:20.068 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:20.068 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:19.0.0-191.g6d933094.el9 ceph 170 k 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 170 k 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 490 k 2023-12-16T11:08:20.069 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-12-16T11:08:20.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2023-12-16T11:08:20.588 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-19.0.0-191.g6d933094.el9.x86_64.rpm 328 kB/s | 170 kB 00:00 2023-12-16T11:08:20.589 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:20.589 INFO:teuthology.orchestra.run.smithi087.stdout:Total 327 kB/s | 170 kB 00:00 2023-12-16T11:08:20.589 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-12-16T11:08:20.597 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-12-16T11:08:20.597 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-12-16T11:08:20.645 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:36 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:20.653 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-12-16T11:08:20.654 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-12-16T11:08:20.791 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:19.0.0-191.g6d933094.el9 ceph 85 k 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:20.792 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 85 k 2023-12-16T11:08:20.793 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 231 k 2023-12-16T11:08:20.793 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:20.803 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-12-16T11:08:20.911 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:21.138 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-19.0.0-191.g6d933094.el9.x86_64.rpm 246 kB/s | 85 kB 00:00 2023-12-16T11:08:21.139 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:21.139 INFO:teuthology.orchestra.run.smithi160.stdout:Total 245 kB/s | 85 kB 00:00 2023-12-16T11:08:21.139 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:21.146 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:21.146 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:21.200 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:21.200 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:21.340 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:21.365 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:21.444 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:21.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:21.702 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:21.702 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-12-16T11:08:21.702 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:21.702 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:21.702 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:21.799 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:21.901 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2023-12-16T11:08:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:22.082 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:22.082 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:22.083 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:22.083 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:22.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2023-12-16T11:08:22.474 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:44 ago on Sat 16 Dec 2023 11:06:38 AM UTC. 2023-12-16T11:08:22.578 INFO:teuthology.orchestra.run.smithi087.stdout:Package rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:22.631 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-12-16T11:08:22.632 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-12-16T11:08:22.632 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-12-16T11:08:22.706 DEBUG:teuthology.parallel:result is None 2023-12-16T11:08:22.866 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:38 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:23.016 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:19.0.0-191.g6d933094.el9 ceph 3.0 M 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:23.017 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.0 M 2023-12-16T11:08:23.018 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 12 M 2023-12-16T11:08:23.018 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:23.558 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-19.0.0-191.g6d933094.el9.x86_64.rpm 5.6 MB/s | 3.0 MB 00:00 2023-12-16T11:08:23.558 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:23.559 INFO:teuthology.orchestra.run.smithi160.stdout:Total 5.6 MB/s | 3.0 MB 00:00 2023-12-16T11:08:23.559 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:23.567 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:23.568 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:23.641 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:23.641 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:24.072 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:24.142 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T11:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T11:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T11:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T11:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:25.330 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:25.330 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:25.330 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:25.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2023-12-16T11:08:26.171 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:42 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:26.317 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:26.318 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:26.318 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:19.0.0-191.g6d933094.el9 ceph 170 k 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 170 k 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 490 k 2023-12-16T11:08:26.319 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-12-16T11:08:26.661 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-19.0.0-191.g6d933094.el9.x86_64.rpm 499 kB/s | 170 kB 00:00 2023-12-16T11:08:26.661 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-12-16T11:08:26.661 INFO:teuthology.orchestra.run.smithi160.stdout:Total 496 kB/s | 170 kB 00:00 2023-12-16T11:08:26.662 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-12-16T11:08:26.669 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-12-16T11:08:26.669 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-12-16T11:08:26.724 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-12-16T11:08:26.724 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-12-16T11:08:26.880 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-12-16T11:08:26.983 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:27.438 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:27.789 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 1/1 2023-12-16T11:08:27.789 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:27.789 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-12-16T11:08:27.789 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:19.0.0-191.g6d933094.el9.x86_64 2023-12-16T11:08:27.789 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:27.790 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:28.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2023-12-16T11:08:28.608 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:44 ago on Sat 16 Dec 2023 11:06:44 AM UTC. 2023-12-16T11:08:28.706 INFO:teuthology.orchestra.run.smithi160.stdout:Package rbd-mirror-2:19.0.0-191.g6d933094.el9.x86_64 is already installed. 2023-12-16T11:08:28.756 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-12-16T11:08:28.757 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-12-16T11:08:28.758 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-12-16T11:08:28.823 DEBUG:teuthology.parallel:result is None 2023-12-16T11:08:28.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T11:08:29.051 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T11:08:29.076 INFO:teuthology.orchestra.run.smithi087.stdout:19.0.0-191.g6d933094.el9 2023-12-16T11:08:29.076 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-191.g6d933094.el9 2023-12-16T11:08:29.076 INFO:teuthology.task.install:The correct ceph version 19.0.0-191.g6d933094 is installed. 2023-12-16T11:08:29.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T11:08:29.304 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T11:08:29.325 INFO:teuthology.orchestra.run.smithi160.stdout:19.0.0-191.g6d933094.el9 2023-12-16T11:08:29.325 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-191.g6d933094.el9 2023-12-16T11:08:29.325 INFO:teuthology.task.install:The correct ceph version 19.0.0-191.g6d933094 is installed. 2023-12-16T11:08:29.326 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-16T11:08:29.326 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:29.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T11:08:29.358 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:29.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T11:08:29.393 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-16T11:08:29.393 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:29.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T11:08:29.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T11:08:29.497 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:29.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T11:08:29.525 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T11:08:29.592 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-16T11:08:29.593 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:29.593 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T11:08:29.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T11:08:29.690 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:29.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T11:08:29.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T11:08:29.785 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-16T11:08:29.786 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:29.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T11:08:29.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T11:08:29.881 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:29.881 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T11:08:29.908 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T11:08:29.977 INFO:teuthology.run_tasks:Running task ceph... 2023-12-16T11:08:30.043 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-16T11:08:30.043 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2023-12-16T11:08:30.046 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /var/log/ceph 2023-12-16T11:08:30.073 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-16T11:08:30.073 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-16T11:08:30.113 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-16T11:08:30.141 INFO:tasks.ceph:Creating extra log directories... 2023-12-16T11:08:30.141 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-16T11:08:30.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-16T11:08:30.226 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-16T11:08:30.227 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-16T11:08:30.227 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7493988', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2970, 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPkeJ1senFIzDxS3+j0Z32wtuNeGa/Dm2hwzGwxTXgWXVMGTT5Ri8kCA/tcuyDVlebct8GeVWwqpOqLgBW1Myw4=', 'smithi160.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjmBWJ/0sPWKHpp3zLV72klfKcofmL77fVFaUFRdBAiT3e1l9nCNfxDLV1niDNP2m9XRBvQQUBkDRhoK3L5IRQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-16_00:13:26', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-16T11:08:30.227 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-16T11:08:30.250 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-16T11:08:30.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T11:08:30.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T11:08:30.332 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:30.332 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T11:08:30.384 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T11:08:30.385 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2023-12-16T11:08:30.439 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T11:08:30.439 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:08:30.439 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-16T11:08:30.439 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:08:30.439 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:08:30.439 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-16 11:08:18.249364290 +0000 2023-12-16T11:08:30.440 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-16 11:05:50.070353849 +0000 2023-12-16T11:08:30.440 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-16 11:05:50.070353849 +0000 2023-12-16T11:08:30.440 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2023-12-16 11:05:50.070353849 +0000 2023-12-16T11:08:30.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T11:08:30.504 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-16T11:08:30.504 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-16T11:08:30.504 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000206179 s, 2.5 MB/s 2023-12-16T11:08:30.505 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T11:08:30.561 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2023-12-16T11:08:30.616 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T11:08:30.616 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:08:30.616 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-16T11:08:30.616 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:08:30.616 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:08:30.616 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-16 11:08:18.250364268 +0000 2023-12-16T11:08:30.616 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-16 11:05:50.583342362 +0000 2023-12-16T11:08:30.617 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-16 11:05:50.583342362 +0000 2023-12-16T11:08:30.617 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2023-12-16 11:05:50.583342362 +0000 2023-12-16T11:08:30.617 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T11:08:30.682 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-16T11:08:30.682 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-16T11:08:30.682 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.00021156 s, 2.4 MB/s 2023-12-16T11:08:30.683 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T11:08:30.738 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-16 11:08:18.250364268 +0000 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-16 11:05:51.065331571 +0000 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-16 11:05:51.065331571 +0000 2023-12-16T11:08:30.793 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2023-12-16 11:05:51.065331571 +0000 2023-12-16T11:08:30.794 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T11:08:30.858 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-16T11:08:30.858 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-16T11:08:30.858 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000183697 s, 2.8 MB/s 2023-12-16T11:08:30.860 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T11:08:30.915 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-12-16 11:08:18.250364268 +0000 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-12-16 11:05:51.549320734 +0000 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-12-16 11:05:51.549320734 +0000 2023-12-16T11:08:30.970 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: 2023-12-16 11:05:51.549320734 +0000 2023-12-16T11:08:30.970 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T11:08:31.034 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-12-16T11:08:31.035 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-12-16T11:08:31.035 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000226286 s, 2.3 MB/s 2023-12-16T11:08:31.036 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T11:08:31.091 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:08:31.092 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-16T11:08:31.092 INFO:tasks.ceph:Generating config... 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client] log to stderr = False 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.093 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.094 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.094 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.094 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-16T11:08:31.094 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-16T11:08:31.094 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-16T11:08:31.094 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-16T11:08:31.094 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-16T11:08:31.094 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-16T11:08:31.094 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-16T11:08:31.094 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-16T11:08:31.094 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-12-16T11:08:31.094 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-12-16T11:08:31.094 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-16T11:08:31.095 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-16T11:08:31.095 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-16T11:08:31.096 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-16T11:08:31.184 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.keyring 2023-12-16T11:08:31.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-16T11:08:31.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-16T11:08:31.296 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.87')] 2023-12-16T11:08:31.296 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.87', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-16T11:08:31.297 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:31.297 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T11:08:31.351 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.87 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:31.429 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:31.429 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid f15fcecc-2053-424b-9728-3322fd59761d 2023-12-16T11:08:31.429 INFO:teuthology.orchestra.run.smithi087.stdout:setting min_mon_release = pacific 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:fsid f15fcecc-2053-424b-9728-3322fd59761d 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2023-12-16T11:08:31.432068+0000 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:created 2023-12-16T11:08:31.432068+0000 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 16 (pacific) 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 1 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2023-12-16T11:08:31.430 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-16T11:08:31.431 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T11:08:31.486 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID f15fcecc-2053-424b-9728-3322fd59761d... 2023-12-16T11:08:31.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-16T11:08:31.529 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2023-12-16T11:08:31.577 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2023-12-16T11:08:31.578 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2023-12-16T11:08:31.579 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = 172.21.15.87 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on pool no app = False 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = f15fcecc-2053-424b-9728-3322fd59761d 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.580 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.581 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: bluefs allocator = hybrid 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore allocator = hybrid 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 1/20 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 1/20 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-16T11:08:31.582 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 4/10 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:08:31.583 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T11:08:31.584 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2023-12-16T11:08:31.585 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd = 20 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd_mirror = 15 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: log to stderr = False 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: rbd default features = 125 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: rbd mirror image policy type = simple 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.0] 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.1] 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.2] 2023-12-16T11:08:31.586 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.3] 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2023-12-16T11:08:31.587 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T11:08:31.588 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.589 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.87 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon client directed command retry = 5 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on pool no app = False 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: ms inject socket failures = 5000 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = f15fcecc-2053-424b-9728-3322fd59761d 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2023-12-16T11:08:31.590 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd mclock profile = high_recovery_ops 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug trim objects = true 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2023-12-16T11:08:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: bluefs allocator = hybrid 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore allocator = hybrid 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore block size = 96636764160 2023-12-16T11:08:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore fsck on mount = True 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug bluefs = 1/20 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug bluestore = 1/20 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 20 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug rocksdb = 4/10 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd full ratio = 0.9 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd nearfull ratio = 0.8 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: osd failsafe full ratio = 0.95 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: osd objectstore = bluestore 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2023-12-16T11:08:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2023-12-16T11:08:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: # 1m isn't quite enough 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: mon_down_mkfs_grace = 2m 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_filestore_osds = false 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:08:31.595 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: debug rbd = 20 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: debug rbd_mirror = 15 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: log to stderr = False 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: rbd default features = 125 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: rbd mirror image policy type = simple 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.0] 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.1] 2023-12-16T11:08:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.597 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.2] 2023-12-16T11:08:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.597 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.3] 2023-12-16T11:08:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:08:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:08:31.599 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-16T11:08:31.599 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-16T11:08:31.694 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-16T11:08:31.694 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:31.694 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-16T11:08:31.709 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:31.709 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-16T11:08:31.764 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:08:31.764 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:31.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-16T11:08:31.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-16T11:08:31.844 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:31.844 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:31.898 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:08:31.898 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:31.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-16T11:08:31.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-16T11:08:31.943 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:31.943 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:32.000 INFO:tasks.ceph:Setting up mon nodes... 2023-12-16T11:08:32.000 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-16T11:08:32.000 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-16T11:08:32.054 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-16T11:08:32.057 INFO:tasks.ceph:Setting up mds nodes... 2023-12-16T11:08:32.057 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-16T11:08:32.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-16T11:08:32.099 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-16T11:08:32.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-16T11:08:32.170 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-16T11:08:32.186 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-16T11:08:32.229 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-16T11:08:32.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-16T11:08:32.328 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-16T11:08:32.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-16T11:08:32.425 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-16T11:08:32.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-16T11:08:32.522 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-16T11:08:32.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-16T11:08:32.619 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-16T11:08:32.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-16T11:08:32.717 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-16T11:08:32.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-16T11:08:32.815 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-16T11:08:32.830 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-16T11:08:32.831 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-16T11:08:32.831 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-16T11:08:32.860 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:08:32.860 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-16T11:08:32.860 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:08:32.860 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-16T11:08:32.935 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T11:08:32.935 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T11:08:32.935 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T11:08:32.935 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T11:08:32.936 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T11:08:32.936 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-12-16T11:08:32.936 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T11:08:32.936 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T11:08:32.936 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T11:08:32.936 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T11:08:33.448 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-12-16T11:08:33.450 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-12-16T11:08:33.451 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-16T11:08:33.603 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-16T11:08:33.639 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-16T11:08:33.707 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:08:33.708 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-16T11:08:33.708 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:08:33.708 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T11:08:33.782 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T11:08:34.231 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-12-16T11:08:34.233 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-12-16T11:08:34.233 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-16T11:08:34.330 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-16T11:08:34.361 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-16T11:08:34.429 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:08:34.429 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-16T11:08:34.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:08:34.429 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-16T11:08:34.502 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T11:08:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T11:08:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T11:08:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T11:08:34.502 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T11:08:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-12-16T11:08:34.502 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T11:08:34.503 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T11:08:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T11:08:34.503 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T11:08:34.993 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-12-16T11:08:34.995 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-12-16T11:08:34.995 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-16T11:08:35.132 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-16T11:08:35.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:35.262 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:35.263+0000 7f1952069500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-16T11:08:35.268 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:35.269+0000 7f1952069500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-16T11:08:35.268 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:35.269+0000 7f1952069500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-16T11:08:35.268 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:35.269+0000 7f1952069500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-16T11:08:35.268 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:35.269+0000 7f1952069500 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-16T11:08:35.275 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:35.277+0000 7f1952069500 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-16T11:08:38.128 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-16T11:08:38.156 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:38.244 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:38.245+0000 7fcfd6e0f500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-16T11:08:38.244 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:38.245+0000 7fcfd6e0f500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-16T11:08:38.244 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:38.245+0000 7fcfd6e0f500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-16T11:08:38.245 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:38.245+0000 7fcfd6e0f500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-16T11:08:38.245 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:38.245+0000 7fcfd6e0f500 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-16T11:08:38.245 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:38.246+0000 7fcfd6e0f500 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-16T11:08:41.114 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-16T11:08:41.143 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:41.230 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:41.231+0000 7fb973d74500 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-16T11:08:41.230 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:41.231+0000 7fb973d74500 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-16T11:08:41.230 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:41.231+0000 7fb973d74500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-16T11:08:41.230 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:41.231+0000 7fb973d74500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-16T11:08:41.230 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:41.231+0000 7fb973d74500 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-16T11:08:41.231 INFO:teuthology.orchestra.run.smithi087.stderr:2023-12-16T11:08:41.232+0000 7fb973d74500 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-16T11:08:44.081 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-16T11:08:44.111 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-16T11:08:44.112 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:44.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-16T11:08:44.182 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:44.182 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-16T11:08:44.252 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:44.252 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-16T11:08:44.320 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:44.320 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-16T11:08:44.389 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:44.389 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-16T11:08:44.444 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.444 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-16T11:08:44.463 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.463 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-16T11:08:44.521 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.521 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-16T11:08:44.579 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.580 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-16T11:08:44.637 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.637 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-16T11:08:44.695 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.695 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-16T11:08:44.753 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.753 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-16T11:08:44.811 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:08:44.811 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-16T11:08:44.867 INFO:tasks.ceph:Adding keys to all mons... 2023-12-16T11:08:44.867 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-16T11:08:44.896 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lKBVqAxAAKhnMoJPPAi/prULf9Rg18w== 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAzhX1lOMvHDxAACFEiXWv+RhKJLD8OaFB56A== 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQA2hX1lBWSzDhAAfTKIVwsHe8eMRVFosOqpTg== 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQA5hX1lHyLbDRAAO4OB2pONXH63DPnuBZM6CQ== 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lFOwUBhAAqMfmi0yUNFnox6tBp5p3Eg== 2023-12-16T11:08:44.897 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror] 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lzf9TChAAGVGtbmKJfWvxN3ZD/jpM3Q== 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.0] 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lf2nWDRAAxkVkyfgyQar2PwwNxT0taw== 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.1] 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lWv21ExAAx1ncs5LGBW8v3Z2r7T0S2Q== 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.2] 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lwXaHGRAAwpRP9QyVpj0FQYcUWL78uA== 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.3] 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lSOlMHxAAAaJDUOPbbj0+tpWi5J1S4g== 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.4] 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lnIMRJRAA26XZfx0Q04OzFlA7tj7Fsw== 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.5] 2023-12-16T11:08:44.898 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1ldWPtKhAA/tHlzYvLk+Z7Jq38R8vArg== 2023-12-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.6] 2023-12-16T11:08:44.899 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAwhX1lrc3HMBAAZCEeYUOC+3l6YKlCCleoVg== 2023-12-16T11:08:44.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T11:08:44.996 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T11:08:45.049 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T11:08:45.100 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T11:08:45.152 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.206 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.261 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.316 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.480 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.534 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.589 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:08:45.644 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-16T11:08:45.644 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-16T11:08:45.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-16T11:08:45.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-16T11:08:45.794 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-16T11:08:45.849 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-16T11:08:45.850 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-16T11:08:45.850 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-16T11:08:45.892 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-16T11:08:45.892 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-16T11:08:45.892 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-16T11:08:45.892 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-16T11:08:45.895 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-16T11:08:45.895 INFO:tasks.ceph:Setting crush tunables to default 2023-12-16T11:08:45.895 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-16T11:08:46.083 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2023-12-16T11:08:46.094 INFO:tasks.ceph:check_enable_crimson: False 2023-12-16T11:08:46.094 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-16T11:08:46.094 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:46.094 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-16T11:08:46.120 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:46.120 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-16T11:08:46.187 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:08:46.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-16T11:08:46.255 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster1 osd new 48068439-5b2f-425e-80db-df998bcb0fca 0 2023-12-16T11:08:46.473 INFO:teuthology.orchestra.run.smithi087.stdout:0 2023-12-16T11:08:46.485 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster1 osd new df567ad0-a71a-478c-9675-70c482bc93d5 1 2023-12-16T11:08:46.642 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:46.643+0000 7f1232e52040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T11:08:46.665 INFO:teuthology.orchestra.run.smithi087.stdout:1 2023-12-16T11:08:46.676 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster1 osd new 5a05331c-9130-4466-92a6-6661928b0926 2 2023-12-16T11:08:46.696 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:46.697+0000 7f1232e52040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T11:08:46.856 INFO:teuthology.orchestra.run.smithi087.stdout:2 2023-12-16T11:08:46.867 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-16T11:08:46.867 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-16T11:08:46.909 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-16T11:08:46.909 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-16T11:08:46.909 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-16T11:08:46.911 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-16T11:08:46.911 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-16T11:08:46.911 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-16T11:08:46.913 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-16T11:08:46.913 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-16T11:08:47.116 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:08:47.117 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"fsid":"f15fcecc-2053-424b-9728-3322fd59761d","created":"2023-12-16T11:08:46.007497+0000","modified":"2023-12-16T11:08:46.857634+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T11:08:46.857634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"48068439-5b2f-425e-80db-df998bcb0fca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"df567ad0-a71a-478c-9675-70c482bc93d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a05331c-9130-4466-92a6-6661928b0926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:08:47.128 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-16T11:08:47.128 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-16T11:08:47.300 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.301+0000 7f1232e52040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T11:08:47.352 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.353+0000 7f1232e52040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T11:08:47.397 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.398+0000 7f1232e52040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T11:08:47.430 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-16T11:08:47.453 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.454+0000 7f1232e52040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T11:08:47.496 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.498+0000 7f1232e52040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T11:08:47.506 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:08:47.507+0000 7f98e4a2b500 -1 Falling back to public interface 2023-12-16T11:08:47.507 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:08:47.507+0000 7fcefe00c500 -1 Falling back to public interface 2023-12-16T11:08:47.507 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:08:47.507+0000 7fcefcaa6640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-12-16T11:08:47.507 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:08:47.508+0000 7f806deaf500 -1 Falling back to public interface 2023-12-16T11:08:47.508 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:08:47.509+0000 7f98e303a640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6814/0: (98) Address already in use 2023-12-16T11:08:47.590 INFO:teuthology.misc.health.smithi087.stdout: 2023-12-16T11:08:47.590 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":5,"fsid":"f15fcecc-2053-424b-9728-3322fd59761d","created":"2023-12-16T11:08:46.007497+0000","modified":"2023-12-16T11:08:46.857634+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T11:08:46.857634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"48068439-5b2f-425e-80db-df998bcb0fca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"df567ad0-a71a-478c-9675-70c482bc93d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a05331c-9130-4466-92a6-6661928b0926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:08:47.600 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-16T11:08:47.605 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.607+0000 7f1232e52040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T11:08:47.677 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.678+0000 7f1232e52040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T11:08:47.721 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.722+0000 7f1232e52040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T11:08:47.773 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.774+0000 7f1232e52040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T11:08:47.821 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:47.822+0000 7f1232e52040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T11:08:48.197 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.198+0000 7f1232e52040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T11:08:48.332 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.333+0000 7f1232e52040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T11:08:48.439 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.441+0000 7f1232e52040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T11:08:48.484 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.485+0000 7f1232e52040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T11:08:48.579 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.580+0000 7f1232e52040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T11:08:48.675 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.676+0000 7f1232e52040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T11:08:48.814 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.815+0000 7f1232e52040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T11:08:48.856 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.857+0000 7f1232e52040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T11:08:48.997 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:48.998+0000 7f1232e52040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T11:08:49.803 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:49.804+0000 7f1232e52040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T11:08:50.076 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:50.077+0000 7f1232e52040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T11:08:50.125 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:50.127+0000 7f1232e52040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T11:08:50.225 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:50.226+0000 7f1232e52040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T11:08:51.010 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:08:51.011+0000 7fcefe00c500 -1 osd.2 0 log_to_monitors true 2023-12-16T11:08:51.056 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:08:51.057+0000 7f806deaf500 -1 osd.0 0 log_to_monitors true 2023-12-16T11:08:51.069 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:08:51.071+0000 7f98e4a2b500 -1 osd.1 0 log_to_monitors true 2023-12-16T11:08:51.244 INFO:tasks.ceph.cluster1.mgr.x.smithi087.stderr:2023-12-16T11:08:51.246+0000 7f11d4251640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-16T11:08:53.903 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-16T11:08:54.270 INFO:teuthology.misc.health.smithi087.stdout: 2023-12-16T11:08:54.270 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":7,"fsid":"f15fcecc-2053-424b-9728-3322fd59761d","created":"2023-12-16T11:08:46.007497+0000","modified":"2023-12-16T11:08:52.977618+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T11:08:46.857634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"48068439-5b2f-425e-80db-df998bcb0fca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"df567ad0-a71a-478c-9675-70c482bc93d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5a05331c-9130-4466-92a6-6661928b0926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:08:54.283 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-16T11:08:54.776 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:08:54.776+0000 7f8069e3d640 -1 osd.0 0 waiting for initial osdmap 2023-12-16T11:08:54.785 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:08:54.786+0000 7f98e0021640 -1 osd.1 0 waiting for initial osdmap 2023-12-16T11:08:54.790 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:08:54.792+0000 7f8061c64640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T11:08:54.798 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:08:54.800+0000 7f98d7e48640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T11:08:54.810 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:08:54.812+0000 7fcef9a8d640 -1 osd.2 0 waiting for initial osdmap 2023-12-16T11:08:54.825 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:08:54.827+0000 7fcef18b4640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T11:09:00.586 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-16T11:09:00.864 INFO:teuthology.misc.health.smithi087.stdout: 2023-12-16T11:09:00.865 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":11,"fsid":"f15fcecc-2053-424b-9728-3322fd59761d","created":"2023-12-16T11:08:46.007497+0000","modified":"2023-12-16T11:08:58.982538+0000","last_up_change":"2023-12-16T11:08:55.788211+0000","last_in_change":"2023-12-16T11:08:46.857634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T11:08:56.289896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48068439-5b2f-425e-80db-df998bcb0fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6807","nonce":2153554517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6813","nonce":2153554517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6823","nonce":2153554517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6819","nonce":2153554517}]},"public_addr":"172.21.15.87:6807/2153554517","cluster_addr":"172.21.15.87:6813/2153554517","heartbeat_back_addr":"172.21.15.87:6823/2153554517","heartbeat_front_addr":"172.21.15.87:6819/2153554517","state":["exists","up"]},{"osd":1,"uuid":"df567ad0-a71a-478c-9675-70c482bc93d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6801","nonce":3512712324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6806","nonce":3512712324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6818","nonce":3512712324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6811","nonce":3512712324}]},"public_addr":"172.21.15.87:6801/3512712324","cluster_addr":"172.21.15.87:6806/3512712324","heartbeat_back_addr":"172.21.15.87:6818/3512712324","heartbeat_front_addr":"172.21.15.87:6811/3512712324","state":["exists","up"]},{"osd":2,"uuid":"5a05331c-9130-4466-92a6-6661928b0926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6805","nonce":1086837511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6812","nonce":1086837511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6822","nonce":1086837511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6817","nonce":1086837511}]},"public_addr":"172.21.15.87:6805/1086837511","cluster_addr":"172.21.15.87:6812/1086837511","heartbeat_back_addr":"172.21.15.87:6822/1086837511","heartbeat_front_addr":"172.21.15.87:6817/1086837511","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.017629+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.043425+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.015849+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:00.876 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-16T11:09:00.876 INFO:tasks.ceph:Creating RBD pool 2023-12-16T11:09:00.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-16T11:09:01.993 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'rbd' created 2023-12-16T11:09:02.009 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-16T11:09:02.993 INFO:teuthology.orchestra.run.smithi087.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-16T11:09:03.014 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-16T11:09:03.014 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-16T11:09:03.014 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-16T11:09:03.015 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-16T11:09:03.015 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-16T11:09:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:03.354 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6824","nonce":254501180},{"type":"v1","addr":"172.21.15.87:6825","nonce":254501180}]},"active_addr":"172.21.15.87:6825/254501180","active_change":"2023-12-16T11:08:50.240980+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":1012351972}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":157581660}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":4058222799}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":548621646}]}]} 2023-12-16T11:09:03.356 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-16T11:09:03.357 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-16T11:09:03.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-16T11:09:03.634 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:03.634 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":13,"fsid":"f15fcecc-2053-424b-9728-3322fd59761d","created":"2023-12-16T11:08:46.007497+0000","modified":"2023-12-16T11:09:02.995047+0000","last_up_change":"2023-12-16T11:08:55.788211+0000","last_in_change":"2023-12-16T11:08:46.857634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T11:08:56.289896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-16T11:09:01.171025+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48068439-5b2f-425e-80db-df998bcb0fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6807","nonce":2153554517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6813","nonce":2153554517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6823","nonce":2153554517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6819","nonce":2153554517}]},"public_addr":"172.21.15.87:6807/2153554517","cluster_addr":"172.21.15.87:6813/2153554517","heartbeat_back_addr":"172.21.15.87:6823/2153554517","heartbeat_front_addr":"172.21.15.87:6819/2153554517","state":["exists","up"]},{"osd":1,"uuid":"df567ad0-a71a-478c-9675-70c482bc93d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6801","nonce":3512712324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6806","nonce":3512712324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6818","nonce":3512712324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6811","nonce":3512712324}]},"public_addr":"172.21.15.87:6801/3512712324","cluster_addr":"172.21.15.87:6806/3512712324","heartbeat_back_addr":"172.21.15.87:6818/3512712324","heartbeat_front_addr":"172.21.15.87:6811/3512712324","state":["exists","up"]},{"osd":2,"uuid":"5a05331c-9130-4466-92a6-6661928b0926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6805","nonce":1086837511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6812","nonce":1086837511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6822","nonce":1086837511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6817","nonce":1086837511}]},"public_addr":"172.21.15.87:6805/1086837511","cluster_addr":"172.21.15.87:6812/1086837511","heartbeat_back_addr":"172.21.15.87:6822/1086837511","heartbeat_front_addr":"172.21.15.87:6817/1086837511","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.017629+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.043425+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.015849+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:03.646 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-16T11:09:03.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-16T11:09:03.925 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:03.925 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":13,"fsid":"f15fcecc-2053-424b-9728-3322fd59761d","created":"2023-12-16T11:08:46.007497+0000","modified":"2023-12-16T11:09:02.995047+0000","last_up_change":"2023-12-16T11:08:55.788211+0000","last_in_change":"2023-12-16T11:08:46.857634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T11:08:56.289896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-16T11:09:01.171025+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48068439-5b2f-425e-80db-df998bcb0fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6807","nonce":2153554517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6813","nonce":2153554517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6823","nonce":2153554517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6819","nonce":2153554517}]},"public_addr":"172.21.15.87:6807/2153554517","cluster_addr":"172.21.15.87:6813/2153554517","heartbeat_back_addr":"172.21.15.87:6823/2153554517","heartbeat_front_addr":"172.21.15.87:6819/2153554517","state":["exists","up"]},{"osd":1,"uuid":"df567ad0-a71a-478c-9675-70c482bc93d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6801","nonce":3512712324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6806","nonce":3512712324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6818","nonce":3512712324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6811","nonce":3512712324}]},"public_addr":"172.21.15.87:6801/3512712324","cluster_addr":"172.21.15.87:6806/3512712324","heartbeat_back_addr":"172.21.15.87:6818/3512712324","heartbeat_front_addr":"172.21.15.87:6811/3512712324","state":["exists","up"]},{"osd":2,"uuid":"5a05331c-9130-4466-92a6-6661928b0926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6805","nonce":1086837511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6812","nonce":1086837511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6822","nonce":1086837511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6817","nonce":1086837511}]},"public_addr":"172.21.15.87:6805/1086837511","cluster_addr":"172.21.15.87:6812/1086837511","heartbeat_back_addr":"172.21.15.87:6822/1086837511","heartbeat_front_addr":"172.21.15.87:6817/1086837511","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.017629+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.043425+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.015849+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:03.937 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-16T11:09:03.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-16T11:09:03.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-16T11:09:04.077 INFO:teuthology.orchestra.run.smithi087.stdout:34359738374 2023-12-16T11:09:04.078 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-16T11:09:04.095 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-12-16T11:09:04.095 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-16T11:09:04.097 INFO:teuthology.orchestra.run.smithi087.stdout:34359738373 2023-12-16T11:09:04.097 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-16T11:09:04.395 INFO:teuthology.orchestra.run.smithi087.stdout:34359738373 2023-12-16T11:09:04.408 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738373 for osd.2 2023-12-16T11:09:04.432 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-12-16T11:09:04.444 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2023-12-16T11:09:04.455 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-12-16T11:09:04.467 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.0 2023-12-16T11:09:05.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-16T11:09:05.445 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-16T11:09:05.468 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-16T11:09:05.717 INFO:teuthology.orchestra.run.smithi087.stdout:34359738374 2023-12-16T11:09:05.730 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2023-12-16T11:09:05.730 DEBUG:teuthology.parallel:result is None 2023-12-16T11:09:05.753 INFO:teuthology.orchestra.run.smithi087.stdout:34359738373 2023-12-16T11:09:05.766 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2023-12-16T11:09:05.766 DEBUG:teuthology.parallel:result is None 2023-12-16T11:09:05.789 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-12-16T11:09:05.801 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2023-12-16T11:09:05.801 DEBUG:teuthology.parallel:result is None 2023-12-16T11:09:05.801 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-16T11:09:05.802 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-16T11:09:06.119 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:06.120 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-16T11:09:06.131 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-16T11:09:04.246767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82568,"kb_used_data":1500,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032952,"statfs":{"total":289910292480,"available":289825742848,"internally_reserved":0,"allocated":1536000,"data_stored":1277438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.250774"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.007322+0000","last_change":"2023-12-16T11:09:03.005273+0000","last_active":"2023-12-16T11:09:03.007322+0000","last_peered":"2023-12-16T11:09:03.007322+0000","last_clean":"2023-12-16T11:09:03.007322+0000","last_became_active":"2023-12-16T11:09:03.004893+0000","last_became_peered":"2023-12-16T11:09:03.004893+0000","last_unstale":"2023-12-16T11:09:03.007322+0000","last_undegraded":"2023-12-16T11:09:03.007322+0000","last_fullsized":"2023-12-16T11:09:03.007322+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:36:54.164666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.007047+0000","last_change":"2023-12-16T11:09:03.005695+0000","last_active":"2023-12-16T11:09:03.007047+0000","last_peered":"2023-12-16T11:09:03.007047+0000","last_clean":"2023-12-16T11:09:03.007047+0000","last_became_active":"2023-12-16T11:09:03.005445+0000","last_became_peered":"2023-12-16T11:09:03.005445+0000","last_unstale":"2023-12-16T11:09:03.007047+0000","last_undegraded":"2023-12-16T11:09:03.007047+0000","last_fullsized":"2023-12-16T11:09:03.007047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:50:23.610841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.007040+0000","last_change":"2023-12-16T11:09:03.005110+0000","last_active":"2023-12-16T11:09:03.007040+0000","last_peered":"2023-12-16T11:09:03.007040+0000","last_clean":"2023-12-16T11:09:03.007040+0000","last_became_active":"2023-12-16T11:09:03.004818+0000","last_became_peered":"2023-12-16T11:09:03.004818+0000","last_unstale":"2023-12-16T11:09:03.007040+0000","last_undegraded":"2023-12-16T11:09:03.007040+0000","last_fullsized":"2023-12-16T11:09:03.007040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T18:37:53.473566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.006849+0000","last_change":"2023-12-16T11:09:03.004551+0000","last_active":"2023-12-16T11:09:03.006849+0000","last_peered":"2023-12-16T11:09:03.006849+0000","last_clean":"2023-12-16T11:09:03.006849+0000","last_became_active":"2023-12-16T11:09:03.004235+0000","last_became_peered":"2023-12-16T11:09:03.004235+0000","last_unstale":"2023-12-16T11:09:03.006849+0000","last_undegraded":"2023-12-16T11:09:03.006849+0000","last_fullsized":"2023-12-16T11:09:03.006849+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:53:03.232670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:04.002850+0000","last_change":"2023-12-16T11:09:03.006609+0000","last_active":"2023-12-16T11:09:04.002850+0000","last_peered":"2023-12-16T11:09:04.002850+0000","last_clean":"2023-12-16T11:09:04.002850+0000","last_became_active":"2023-12-16T11:09:03.006307+0000","last_became_peered":"2023-12-16T11:09:03.006307+0000","last_unstale":"2023-12-16T11:09:04.002850+0000","last_undegraded":"2023-12-16T11:09:04.002850+0000","last_fullsized":"2023-12-16T11:09:04.002850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:33:31.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.006191+0000","last_change":"2023-12-16T11:09:03.004443+0000","last_active":"2023-12-16T11:09:03.006191+0000","last_peered":"2023-12-16T11:09:03.006191+0000","last_clean":"2023-12-16T11:09:03.006191+0000","last_became_active":"2023-12-16T11:09:03.004159+0000","last_became_peered":"2023-12-16T11:09:03.004159+0000","last_unstale":"2023-12-16T11:09:03.006191+0000","last_undegraded":"2023-12-16T11:09:03.006191+0000","last_fullsized":"2023-12-16T11:09:03.006191+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:39:39.023067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.005848+0000","last_change":"2023-12-16T11:09:03.004009+0000","last_active":"2023-12-16T11:09:03.005848+0000","last_peered":"2023-12-16T11:09:03.005848+0000","last_clean":"2023-12-16T11:09:03.005848+0000","last_became_active":"2023-12-16T11:09:03.003715+0000","last_became_peered":"2023-12-16T11:09:03.003715+0000","last_unstale":"2023-12-16T11:09:03.005848+0000","last_undegraded":"2023-12-16T11:09:03.005848+0000","last_fullsized":"2023-12-16T11:09:03.005848+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:40:00.379166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.007303+0000","last_change":"2023-12-16T11:09:03.006746+0000","last_active":"2023-12-16T11:09:03.007303+0000","last_peered":"2023-12-16T11:09:03.007303+0000","last_clean":"2023-12-16T11:09:03.007303+0000","last_became_active":"2023-12-16T11:09:03.006304+0000","last_became_peered":"2023-12-16T11:09:03.006304+0000","last_unstale":"2023-12-16T11:09:03.007303+0000","last_undegraded":"2023-12-16T11:09:03.007303+0000","last_fullsized":"2023-12-16T11:09:03.007303+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:20:10.203364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-16T11:09:02.000419+0000","last_change":"2023-12-16T11:08:58.062810+0000","last_active":"2023-12-16T11:09:02.000419+0000","last_peered":"2023-12-16T11:09:02.000419+0000","last_clean":"2023-12-16T11:09:02.000419+0000","last_became_active":"2023-12-16T11:08:58.061796+0000","last_became_peered":"2023-12-16T11:08:58.061796+0000","last_unstale":"2023-12-16T11:09:02.000419+0000","last_undegraded":"2023-12-16T11:09:02.000419+0000","last_fullsized":"2023-12-16T11:09:02.000419+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_clean_scrub_stamp":"2023-12-16T11:08:56.977736+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:33:16.597547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":712704,"data_stored":624408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:09:06.132 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-16T11:09:06.407 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:06.408 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-16T11:09:06.419 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-16T11:09:06.247142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82576,"kb_used_data":1508,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032944,"statfs":{"total":289910292480,"available":289825734656,"internally_reserved":0,"allocated":1544192,"data_stored":1280916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.251148"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:05.476326+0000","last_change":"2023-12-16T11:09:03.005273+0000","last_active":"2023-12-16T11:09:05.476326+0000","last_peered":"2023-12-16T11:09:05.476326+0000","last_clean":"2023-12-16T11:09:05.476326+0000","last_became_active":"2023-12-16T11:09:03.004893+0000","last_became_peered":"2023-12-16T11:09:03.004893+0000","last_unstale":"2023-12-16T11:09:05.476326+0000","last_undegraded":"2023-12-16T11:09:05.476326+0000","last_fullsized":"2023-12-16T11:09:05.476326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:41:43.307235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:05.476270+0000","last_change":"2023-12-16T11:09:03.005695+0000","last_active":"2023-12-16T11:09:05.476270+0000","last_peered":"2023-12-16T11:09:05.476270+0000","last_clean":"2023-12-16T11:09:05.476270+0000","last_became_active":"2023-12-16T11:09:03.005445+0000","last_became_peered":"2023-12-16T11:09:03.005445+0000","last_unstale":"2023-12-16T11:09:05.476270+0000","last_undegraded":"2023-12-16T11:09:05.476270+0000","last_fullsized":"2023-12-16T11:09:05.476270+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:17:51.300986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:05.476236+0000","last_change":"2023-12-16T11:09:03.005110+0000","last_active":"2023-12-16T11:09:05.476236+0000","last_peered":"2023-12-16T11:09:05.476236+0000","last_clean":"2023-12-16T11:09:05.476236+0000","last_became_active":"2023-12-16T11:09:03.004818+0000","last_became_peered":"2023-12-16T11:09:03.004818+0000","last_unstale":"2023-12-16T11:09:05.476236+0000","last_undegraded":"2023-12-16T11:09:05.476236+0000","last_fullsized":"2023-12-16T11:09:05.476236+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:13:30.010815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:05.476211+0000","last_change":"2023-12-16T11:09:03.004551+0000","last_active":"2023-12-16T11:09:05.476211+0000","last_peered":"2023-12-16T11:09:05.476211+0000","last_clean":"2023-12-16T11:09:05.476211+0000","last_became_active":"2023-12-16T11:09:03.004235+0000","last_became_peered":"2023-12-16T11:09:03.004235+0000","last_unstale":"2023-12-16T11:09:05.476211+0000","last_undegraded":"2023-12-16T11:09:05.476211+0000","last_fullsized":"2023-12-16T11:09:05.476211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:25:33.320035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:04.002850+0000","last_change":"2023-12-16T11:09:03.006609+0000","last_active":"2023-12-16T11:09:04.002850+0000","last_peered":"2023-12-16T11:09:04.002850+0000","last_clean":"2023-12-16T11:09:04.002850+0000","last_became_active":"2023-12-16T11:09:03.006307+0000","last_became_peered":"2023-12-16T11:09:03.006307+0000","last_unstale":"2023-12-16T11:09:04.002850+0000","last_undegraded":"2023-12-16T11:09:04.002850+0000","last_fullsized":"2023-12-16T11:09:04.002850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:33:31.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:05.616015+0000","last_change":"2023-12-16T11:09:03.004443+0000","last_active":"2023-12-16T11:09:05.616015+0000","last_peered":"2023-12-16T11:09:05.616015+0000","last_clean":"2023-12-16T11:09:05.616015+0000","last_became_active":"2023-12-16T11:09:03.004159+0000","last_became_peered":"2023-12-16T11:09:03.004159+0000","last_unstale":"2023-12-16T11:09:05.616015+0000","last_undegraded":"2023-12-16T11:09:05.616015+0000","last_fullsized":"2023-12-16T11:09:05.616015+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:10:30.195137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:05.616032+0000","last_change":"2023-12-16T11:09:03.004009+0000","last_active":"2023-12-16T11:09:05.616032+0000","last_peered":"2023-12-16T11:09:05.616032+0000","last_clean":"2023-12-16T11:09:05.616032+0000","last_became_active":"2023-12-16T11:09:03.003715+0000","last_became_peered":"2023-12-16T11:09:03.003715+0000","last_unstale":"2023-12-16T11:09:05.616032+0000","last_undegraded":"2023-12-16T11:09:05.616032+0000","last_fullsized":"2023-12-16T11:09:05.616032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:12:11.654608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:05.476240+0000","last_change":"2023-12-16T11:09:03.006746+0000","last_active":"2023-12-16T11:09:05.476240+0000","last_peered":"2023-12-16T11:09:05.476240+0000","last_clean":"2023-12-16T11:09:05.476240+0000","last_became_active":"2023-12-16T11:09:03.006304+0000","last_became_peered":"2023-12-16T11:09:03.006304+0000","last_unstale":"2023-12-16T11:09:05.476240+0000","last_undegraded":"2023-12-16T11:09:05.476240+0000","last_fullsized":"2023-12-16T11:09:05.476240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:20:34.752327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:03.002634+0000","last_change":"2023-12-16T11:08:58.062810+0000","last_active":"2023-12-16T11:09:03.002634+0000","last_peered":"2023-12-16T11:09:03.002634+0000","last_clean":"2023-12-16T11:09:03.002634+0000","last_became_active":"2023-12-16T11:08:58.061796+0000","last_became_peered":"2023-12-16T11:08:58.061796+0000","last_unstale":"2023-12-16T11:09:03.002634+0000","last_undegraded":"2023-12-16T11:09:03.002634+0000","last_fullsized":"2023-12-16T11:09:03.002634+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_clean_scrub_stamp":"2023-12-16T11:08:56.977736+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:33:16.597547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":712704,"data_stored":624408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:09:06.420 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-16T11:09:06.420 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-16T11:09:06.420 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-16T11:09:06.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-16T11:09:06.729 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:06.730 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-16T11:09:06.742 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-16T11:09:06.742 INFO:teuthology.run_tasks:Running task ceph... 2023-12-16T11:09:06.755 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-16T11:09:06.755 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-16T11:09:06.755 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7493988', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2970, 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678', 'targets': {'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPkeJ1senFIzDxS3+j0Z32wtuNeGa/Dm2hwzGwxTXgWXVMGTT5Ri8kCA/tcuyDVlebct8GeVWwqpOqLgBW1Myw4=', 'smithi160.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjmBWJ/0sPWKHpp3zLV72klfKcofmL77fVFaUFRdBAiT3e1l9nCNfxDLV1niDNP2m9XRBvQQUBkDRhoK3L5IRQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-16_00:13:26', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-16T11:09:06.756 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-16T11:09:06.758 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-16T11:09:06.777 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T11:09:06.813 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T11:09:06.848 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:06.848 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T11:09:06.906 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T11:09:06.906 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2023-12-16T11:09:06.962 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-12-16 11:08:24.401794916 +0000 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-12-16 11:05:50.063916328 +0000 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-12-16 11:05:50.063916328 +0000 2023-12-16T11:09:06.963 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2023-12-16 11:05:50.063916328 +0000 2023-12-16T11:09:06.963 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T11:09:07.030 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-12-16T11:09:07.031 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-12-16T11:09:07.031 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000219216 s, 2.3 MB/s 2023-12-16T11:09:07.032 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T11:09:07.089 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2023-12-16T11:09:07.144 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-12-16 11:08:24.401794916 +0000 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-12-16 11:05:50.557908619 +0000 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-12-16 11:05:50.557908619 +0000 2023-12-16T11:09:07.145 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2023-12-16 11:05:50.557908619 +0000 2023-12-16T11:09:07.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T11:09:07.212 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-12-16T11:09:07.213 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-12-16T11:09:07.213 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000196033 s, 2.6 MB/s 2023-12-16T11:09:07.214 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T11:09:07.270 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-12-16 11:08:24.401794916 +0000 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-12-16 11:05:51.047900976 +0000 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-12-16 11:05:51.047900976 +0000 2023-12-16T11:09:07.327 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2023-12-16 11:05:51.047900976 +0000 2023-12-16T11:09:07.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T11:09:07.392 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-12-16T11:09:07.392 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-12-16T11:09:07.392 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000200391 s, 2.6 MB/s 2023-12-16T11:09:07.393 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T11:09:07.449 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2023-12-16T11:09:07.505 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T11:09:07.505 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T11:09:07.505 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-16T11:09:07.505 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T11:09:07.505 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T11:09:07.506 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2023-12-16 11:08:24.402794900 +0000 2023-12-16T11:09:07.506 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2023-12-16 11:05:51.528893473 +0000 2023-12-16T11:09:07.506 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2023-12-16 11:05:51.528893473 +0000 2023-12-16T11:09:07.506 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2023-12-16 11:05:51.528893473 +0000 2023-12-16T11:09:07.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T11:09:07.572 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2023-12-16T11:09:07.572 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2023-12-16T11:09:07.572 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000200387 s, 2.6 MB/s 2023-12-16T11:09:07.573 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T11:09:07.629 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:09:07.629 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-16T11:09:07.629 INFO:tasks.ceph:Generating config... 2023-12-16T11:09:07.630 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-16T11:09:07.630 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client] log to stderr = False 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:07.631 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:07.631 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-16T11:09:07.631 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-16T11:09:07.632 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-16T11:09:07.632 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-16T11:09:07.632 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-16T11:09:07.632 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-16T11:09:07.632 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-16T11:09:07.632 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-16T11:09:07.632 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-16T11:09:07.633 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-16T11:09:07.633 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-16T11:09:07.633 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-16T11:09:07.633 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-16T11:09:07.633 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-16T11:09:07.633 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-16T11:09:07.633 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-16T11:09:07.633 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-16T11:09:07.710 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.keyring 2023-12-16T11:09:07.713 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-16T11:09:07.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-16T11:09:07.824 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.160')] 2023-12-16T11:09:07.824 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.160', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-16T11:09:07.825 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:07.826 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T11:09:07.879 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.160 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:07.958 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:07.958 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: generated fsid fdb8828c-a0b4-49f4-bc4d-e8356adcab23 2023-12-16T11:09:07.958 INFO:teuthology.orchestra.run.smithi160.stdout:setting min_mon_release = pacific 2023-12-16T11:09:07.958 INFO:teuthology.orchestra.run.smithi160.stdout:epoch 0 2023-12-16T11:09:07.958 INFO:teuthology.orchestra.run.smithi160.stdout:fsid fdb8828c-a0b4-49f4-bc4d-e8356adcab23 2023-12-16T11:09:07.959 INFO:teuthology.orchestra.run.smithi160.stdout:last_changed 2023-12-16T11:09:07.960507+0000 2023-12-16T11:09:07.959 INFO:teuthology.orchestra.run.smithi160.stdout:created 2023-12-16T11:09:07.960507+0000 2023-12-16T11:09:07.959 INFO:teuthology.orchestra.run.smithi160.stdout:min_mon_release 16 (pacific) 2023-12-16T11:09:07.959 INFO:teuthology.orchestra.run.smithi160.stdout:election_strategy: 1 2023-12-16T11:09:07.959 INFO:teuthology.orchestra.run.smithi160.stdout:0: [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon.a 2023-12-16T11:09:07.959 INFO:teuthology.orchestra.run.smithi160.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-16T11:09:07.960 DEBUG:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T11:09:08.014 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID fdb8828c-a0b4-49f4-bc4d-e8356adcab23... 2023-12-16T11:09:08.015 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-16T11:09:08.017 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2023-12-16T11:09:08.072 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.073 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.074 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = 172.21.15.160 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on pool no app = False 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = fdb8828c-a0b4-49f4-bc4d-e8356adcab23 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2023-12-16T11:09:08.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd mclock profile = high_recovery_ops 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug trim objects = true 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2023-12-16T11:09:08.076 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: bluefs allocator = hybrid 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore allocator = hybrid 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2023-12-16T11:09:08.077 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 1/20 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 1/20 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 4/10 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-16T11:09:08.078 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:09:08.079 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: # 1m isn't quite enough 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: mon_down_mkfs_grace = 2m 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_filestore_osds = false 2023-12-16T11:09:08.080 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd = 20 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd_mirror = 15 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: log to stderr = False 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: rbd default features = 125 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: rbd mirror image policy type = simple 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.0] 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.081 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.1] 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.2] 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.3] 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.082 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.104 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2023-12-16T11:09:08.104 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.105 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T11:09:08.106 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = 172.21.15.160 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: mon client directed command retry = 5 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on pool no app = False 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: ms inject socket failures = 5000 2023-12-16T11:09:08.107 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = fdb8828c-a0b4-49f4-bc4d-e8356adcab23 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 600 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd mclock profile = high_recovery_ops 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug trim objects = true 2023-12-16T11:09:08.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2023-12-16T11:09:08.109 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: bluefs allocator = hybrid 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore allocator = hybrid 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore block size = 96636764160 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore fsck on mount = True 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: debug bluefs = 1/20 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: debug bluestore = 1/20 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 20 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: debug rocksdb = 4/10 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd full ratio = 0.9 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd nearfull ratio = 0.8 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: osd failsafe full ratio = 0.95 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: osd objectstore = bluestore 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.110 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2023-12-16T11:09:08.111 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: # 1m isn't quite enough 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: mon_down_mkfs_grace = 2m 2023-12-16T11:09:08.112 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_filestore_osds = false 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: debug rbd = 20 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: debug rbd_mirror = 15 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: log to stderr = False 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: rbd default features = 125 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout: rbd mirror image policy type = simple 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2023-12-16T11:09:08.113 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.0] 2023-12-16T11:09:08.114 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.114 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.114 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.1] 2023-12-16T11:09:08.116 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.116 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.116 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.2] 2023-12-16T11:09:08.116 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.116 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.116 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.3] 2023-12-16T11:09:08.117 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-16T11:09:08.117 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-16T11:09:08.118 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-16T11:09:08.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-16T11:09:08.174 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-16T11:09:08.174 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:08.174 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-16T11:09:08.188 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:08.189 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-16T11:09:08.243 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:09:08.243 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:09:08.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-16T11:09:08.244 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-16T11:09:08.285 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:09:08.285 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:08.342 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:09:08.342 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:08.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-16T11:09:08.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-16T11:09:08.382 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:08.382 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:08.436 INFO:tasks.ceph:Setting up mon nodes... 2023-12-16T11:09:08.437 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-16T11:09:08.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-16T11:09:08.495 INFO:teuthology.orchestra.run.smithi087.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-16T11:09:08.497 INFO:tasks.ceph:Setting up mds nodes... 2023-12-16T11:09:08.497 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-16T11:09:08.497 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-16T11:09:08.539 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-16T11:09:08.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-16T11:09:08.596 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-16T11:09:08.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-16T11:09:08.693 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-16T11:09:08.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-16T11:09:08.791 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-16T11:09:08.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-16T11:09:08.889 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-16T11:09:08.905 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-16T11:09:08.987 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-16T11:09:09.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-16T11:09:09.085 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-16T11:09:09.101 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-16T11:09:09.184 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-16T11:09:09.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-16T11:09:09.281 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-16T11:09:09.296 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-16T11:09:09.296 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-16T11:09:09.296 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-16T11:09:09.363 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:09:09.363 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-16T11:09:09.363 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:09:09.363 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T11:09:09.437 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T11:09:09.949 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2023-12-16T11:09:09.950 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2023-12-16T11:09:09.951 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-16T11:09:10.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-16T11:09:10.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-16T11:09:10.188 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:09:10.188 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-16T11:09:10.188 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:09:10.189 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-16T11:09:10.262 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T11:09:10.263 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T11:09:10.717 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2023-12-16T11:09:10.719 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2023-12-16T11:09:10.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-16T11:09:10.812 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-16T11:09:10.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-16T11:09:10.909 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-16T11:09:10.909 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-16T11:09:10.909 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:09:10.910 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T11:09:10.984 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T11:09:11.476 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2023-12-16T11:09:11.478 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2023-12-16T11:09:11.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-16T11:09:11.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-16T11:09:11.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:11.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:11.760+0000 7fc50c94a500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-16T11:09:11.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:11.764+0000 7fc50c94a500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-16T11:09:11.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:11.764+0000 7fc50c94a500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-16T11:09:11.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:11.764+0000 7fc50c94a500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-16T11:09:11.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:11.764+0000 7fc50c94a500 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-16T11:09:11.767 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:11.768+0000 7fc50c94a500 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-16T11:09:14.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-16T11:09:14.652 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:14.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:14.742+0000 7f1f57c1f500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-16T11:09:14.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:14.742+0000 7f1f57c1f500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-16T11:09:14.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:14.742+0000 7f1f57c1f500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-16T11:09:14.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:14.742+0000 7f1f57c1f500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-16T11:09:14.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:14.742+0000 7f1f57c1f500 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-16T11:09:14.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:14.743+0000 7f1f57c1f500 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-16T11:09:17.581 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-16T11:09:17.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:17.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:17.701+0000 7f559bf74500 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-16T11:09:17.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:17.701+0000 7f559bf74500 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-16T11:09:17.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:17.701+0000 7f559bf74500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-16T11:09:17.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:17.701+0000 7f559bf74500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-16T11:09:17.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:17.701+0000 7f559bf74500 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-16T11:09:17.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-12-16T11:09:17.702+0000 7f559bf74500 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-16T11:09:20.549 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-16T11:09:20.579 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-16T11:09:20.580 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:09:20.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-16T11:09:20.612 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:20.612 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-16T11:09:20.648 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:20.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-16T11:09:20.718 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:20.718 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-16T11:09:20.786 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-12-16T11:09:20.786 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-16T11:09:20.805 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:20.806 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-16T11:09:20.841 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:20.842 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-16T11:09:20.897 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:20.897 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-16T11:09:20.952 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:20.952 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-16T11:09:21.006 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:21.006 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-16T11:09:21.062 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:21.062 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-16T11:09:21.116 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:21.116 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-16T11:09:21.172 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:21.172 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-16T11:09:21.226 INFO:tasks.ceph:Adding keys to all mons... 2023-12-16T11:09:21.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr.x] 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1luCmjHRAAetOizHXtwSu7F7CbenXhnw== 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.0] 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBXhX1ldMFoLRAAwo2sy6H0a3Ate64DiTy+XA== 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.1] 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBahX1lnkpALBAAHRy4nfHQw2YRDPUQsQZx7g== 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout:[osd.2] 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBdhX1lG5/cKRAA3OHBpAcKtSbz6gawZeLiMA== 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout:[client.0] 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1luF9QIBAARR/N6a4OnxJBE7+qn2TQ/g== 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror] 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lHI+wIxAA19zSds/Xpr0a3W43OIjwPw== 2023-12-16T11:09:21.292 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.0] 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lesZ7KRAA6LAFqZCdWLTTgTnJwqfiiQ== 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.1] 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lGYpILxAAUyQAZnFZsnlD3A4/g62wVw== 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.2] 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lbegpNRAAFsQDUF2RrMCcOYzwRacPeQ== 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.3] 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1l1hz8OhAAdTv2CENTOBvJrAKiT0FCBA== 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.4] 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBVhX1lJO43BRAAEEn0IA7giX3dWaC6vijIZg== 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.5] 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBVhX1lts8TCxAA7x3sBcKYwasErONhj4AH0w== 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.6] 2023-12-16T11:09:21.293 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBVhX1lohbeEBAAmAgMqxL6DE56c1BsKnjc+Q== 2023-12-16T11:09:21.294 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T11:09:21.387 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T11:09:21.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T11:09:21.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T11:09:21.543 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.763 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.872 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:21.982 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T11:09:22.037 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-16T11:09:22.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-16T11:09:22.064 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-16T11:09:22.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-16T11:09:22.187 DEBUG:teuthology.orchestra.run.smithi160:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-16T11:09:22.241 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-16T11:09:22.241 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-16T11:09:22.242 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-16T11:09:22.283 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-16T11:09:22.283 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-16T11:09:22.284 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-16T11:09:22.284 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-16T11:09:22.287 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-16T11:09:22.287 INFO:tasks.ceph:Setting crush tunables to default 2023-12-16T11:09:22.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-16T11:09:22.479 INFO:teuthology.orchestra.run.smithi160.stderr:adjusted tunables profile to default 2023-12-16T11:09:22.490 INFO:tasks.ceph:check_enable_crimson: False 2023-12-16T11:09:22.490 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-16T11:09:22.490 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:22.490 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-16T11:09:22.517 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:22.517 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-16T11:09:22.584 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:09:22.584 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-16T11:09:22.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster cluster2 osd new 03f7755b-3361-49e2-9bcb-09b6a034ca66 0 2023-12-16T11:09:22.741 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:22.742+0000 7fa36462c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T11:09:22.786 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:22.786+0000 7fa36462c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T11:09:22.869 INFO:teuthology.orchestra.run.smithi160.stdout:0 2023-12-16T11:09:22.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster cluster2 osd new f487a8e6-d5e0-437b-b6ac-145ca95125b4 1 2023-12-16T11:09:23.057 INFO:teuthology.orchestra.run.smithi160.stdout:1 2023-12-16T11:09:23.068 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster cluster2 osd new 9269a1cc-1f25-40e9-bdc7-ff9446d26646 2 2023-12-16T11:09:23.172 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.173+0000 7fa36462c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T11:09:23.215 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.216+0000 7fa36462c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T11:09:23.243 INFO:teuthology.orchestra.run.smithi160.stdout:2 2023-12-16T11:09:23.255 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-16T11:09:23.255 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-16T11:09:23.257 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.257+0000 7fa36462c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T11:09:23.258 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-16T11:09:23.258 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-16T11:09:23.258 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-16T11:09:23.260 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-16T11:09:23.261 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-16T11:09:23.261 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-16T11:09:23.263 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-16T11:09:23.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-16T11:09:23.307 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.308+0000 7fa36462c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T11:09:23.348 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.349+0000 7fa36462c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T11:09:23.434 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:23.434 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":5,"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","created":"2023-12-16T11:09:22.402607+0000","modified":"2023-12-16T11:09:23.245240+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T11:09:23.245240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03f7755b-3361-49e2-9bcb-09b6a034ca66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f487a8e6-d5e0-437b-b6ac-145ca95125b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9269a1cc-1f25-40e9-bdc7-ff9446d26646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:23.445 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-16T11:09:23.446 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-16T11:09:23.446 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.445+0000 7fa36462c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T11:09:23.497 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.498+0000 7fa36462c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T11:09:23.538 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.539+0000 7fa36462c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T11:09:23.586 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.587+0000 7fa36462c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T11:09:23.628 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.629+0000 7fa36462c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T11:09:23.749 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-16T11:09:23.853 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:09:23.854+0000 7ff3fa015500 -1 Falling back to public interface 2023-12-16T11:09:23.857 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:09:23.858+0000 7f1408813500 -1 Falling back to public interface 2023-12-16T11:09:23.861 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:09:23.862+0000 7f6f85d12500 -1 Falling back to public interface 2023-12-16T11:09:23.910 INFO:teuthology.misc.health.smithi160.stdout: 2023-12-16T11:09:23.910 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":5,"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","created":"2023-12-16T11:09:22.402607+0000","modified":"2023-12-16T11:09:23.245240+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T11:09:23.245240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03f7755b-3361-49e2-9bcb-09b6a034ca66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f487a8e6-d5e0-437b-b6ac-145ca95125b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9269a1cc-1f25-40e9-bdc7-ff9446d26646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:23.920 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-16T11:09:23.941 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:23.941+0000 7fa36462c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T11:09:24.070 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.071+0000 7fa36462c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T11:09:24.160 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.160+0000 7fa36462c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T11:09:24.201 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.202+0000 7fa36462c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T11:09:24.296 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.297+0000 7fa36462c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T11:09:24.385 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.386+0000 7fa36462c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T11:09:24.519 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.520+0000 7fa36462c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T11:09:24.560 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.561+0000 7fa36462c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T11:09:24.663 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:24.664+0000 7fa36462c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T11:09:25.257 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:25.258+0000 7fa36462c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T11:09:25.514 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:25.515+0000 7fa36462c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T11:09:25.562 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:25.563+0000 7fa36462c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T11:09:25.661 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:25.661+0000 7fa36462c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T11:09:26.680 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:09:26.681+0000 7fa30520a640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-16T11:09:27.277 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:09:27.278+0000 7ff3fa015500 -1 osd.1 0 log_to_monitors true 2023-12-16T11:09:27.331 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:09:27.332+0000 7f1408813500 -1 osd.2 0 log_to_monitors true 2023-12-16T11:09:27.364 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:09:27.365+0000 7f6f85d12500 -1 osd.0 0 log_to_monitors true 2023-12-16T11:09:30.223 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-16T11:09:30.465 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:09:30.466+0000 7f14040ce640 -1 osd.2 0 waiting for initial osdmap 2023-12-16T11:09:30.481 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:09:30.482+0000 7f13fbef5640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T11:09:30.490 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:09:30.491+0000 7f6f81ca0640 -1 osd.0 0 waiting for initial osdmap 2023-12-16T11:09:30.503 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:09:30.504+0000 7f6f79ac7640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T11:09:30.556 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:09:30.557+0000 7ff3f595a640 -1 osd.1 0 waiting for initial osdmap 2023-12-16T11:09:30.570 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:09:30.571+0000 7ff3ed781640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T11:09:30.607 INFO:teuthology.misc.health.smithi160.stdout: 2023-12-16T11:09:30.607 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":7,"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","created":"2023-12-16T11:09:22.402607+0000","modified":"2023-12-16T11:09:28.681229+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T11:09:23.245240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03f7755b-3361-49e2-9bcb-09b6a034ca66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f487a8e6-d5e0-437b-b6ac-145ca95125b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9269a1cc-1f25-40e9-bdc7-ff9446d26646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:30.619 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-16T11:09:36.922 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-16T11:09:37.205 INFO:teuthology.misc.health.smithi160.stdout: 2023-12-16T11:09:37.205 INFO:teuthology.misc.health.smithi160.stdout:{"epoch":11,"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","created":"2023-12-16T11:09:22.402607+0000","modified":"2023-12-16T11:09:34.506296+0000","last_up_change":"2023-12-16T11:09:31.493461+0000","last_in_change":"2023-12-16T11:09:23.245240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T11:09:31.721605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f7755b-3361-49e2-9bcb-09b6a034ca66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6817","nonce":2985674533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6819","nonce":2985674533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6823","nonce":2985674533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6821","nonce":2985674533}]},"public_addr":"172.21.15.160:6817/2985674533","cluster_addr":"172.21.15.160:6819/2985674533","heartbeat_back_addr":"172.21.15.160:6823/2985674533","heartbeat_front_addr":"172.21.15.160:6821/2985674533","state":["exists","up"]},{"osd":1,"uuid":"f487a8e6-d5e0-437b-b6ac-145ca95125b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6801","nonce":445311465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6803","nonce":445311465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6807","nonce":445311465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6805","nonce":445311465}]},"public_addr":"172.21.15.160:6801/445311465","cluster_addr":"172.21.15.160:6803/445311465","heartbeat_back_addr":"172.21.15.160:6807/445311465","heartbeat_front_addr":"172.21.15.160:6805/445311465","state":["exists","up"]},{"osd":2,"uuid":"9269a1cc-1f25-40e9-bdc7-ff9446d26646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6809","nonce":3852362862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6811","nonce":3852362862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6815","nonce":3852362862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6813","nonce":3852362862}]},"public_addr":"172.21.15.160:6809/3852362862","cluster_addr":"172.21.15.160:6811/3852362862","heartbeat_back_addr":"172.21.15.160:6815/3852362862","heartbeat_front_addr":"172.21.15.160:6813/3852362862","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.346273+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.287905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.360519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:37.213 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-16T11:09:37.213 INFO:tasks.ceph:Creating RBD pool 2023-12-16T11:09:37.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-16T11:09:38.370 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'rbd' created 2023-12-16T11:09:38.388 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-16T11:09:39.371 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-16T11:09:39.390 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-16T11:09:39.390 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-16T11:09:39.390 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-16T11:09:39.391 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-16T11:09:39.391 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-16T11:09:39.720 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:39.730 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6826","nonce":2226385547},{"type":"v1","addr":"172.21.15.87:6827","nonce":2226385547}]},"active_addr":"172.21.15.87:6827/2226385547","active_change":"2023-12-16T11:09:25.676325+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":4034918448}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":828546397}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":1306605934}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":3220905921}]}]} 2023-12-16T11:09:39.732 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-16T11:09:39.732 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-16T11:09:39.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-16T11:09:40.008 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:40.008 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":13,"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","created":"2023-12-16T11:09:22.402607+0000","modified":"2023-12-16T11:09:39.372279+0000","last_up_change":"2023-12-16T11:09:31.493461+0000","last_in_change":"2023-12-16T11:09:23.245240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T11:09:31.721605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-16T11:09:37.508354+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f7755b-3361-49e2-9bcb-09b6a034ca66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6817","nonce":2985674533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6819","nonce":2985674533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6823","nonce":2985674533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6821","nonce":2985674533}]},"public_addr":"172.21.15.160:6817/2985674533","cluster_addr":"172.21.15.160:6819/2985674533","heartbeat_back_addr":"172.21.15.160:6823/2985674533","heartbeat_front_addr":"172.21.15.160:6821/2985674533","state":["exists","up"]},{"osd":1,"uuid":"f487a8e6-d5e0-437b-b6ac-145ca95125b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6801","nonce":445311465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6803","nonce":445311465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6807","nonce":445311465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6805","nonce":445311465}]},"public_addr":"172.21.15.160:6801/445311465","cluster_addr":"172.21.15.160:6803/445311465","heartbeat_back_addr":"172.21.15.160:6807/445311465","heartbeat_front_addr":"172.21.15.160:6805/445311465","state":["exists","up"]},{"osd":2,"uuid":"9269a1cc-1f25-40e9-bdc7-ff9446d26646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6809","nonce":3852362862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6811","nonce":3852362862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6815","nonce":3852362862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6813","nonce":3852362862}]},"public_addr":"172.21.15.160:6809/3852362862","cluster_addr":"172.21.15.160:6811/3852362862","heartbeat_back_addr":"172.21.15.160:6815/3852362862","heartbeat_front_addr":"172.21.15.160:6813/3852362862","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.346273+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.287905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.360519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:40.020 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-16T11:09:40.020 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-16T11:09:40.298 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:40.298 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":13,"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","created":"2023-12-16T11:09:22.402607+0000","modified":"2023-12-16T11:09:39.372279+0000","last_up_change":"2023-12-16T11:09:31.493461+0000","last_in_change":"2023-12-16T11:09:23.245240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T11:09:31.721605+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-16T11:09:37.508354+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f7755b-3361-49e2-9bcb-09b6a034ca66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6816","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6817","nonce":2985674533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6818","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6819","nonce":2985674533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6822","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6823","nonce":2985674533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6820","nonce":2985674533},{"type":"v1","addr":"172.21.15.160:6821","nonce":2985674533}]},"public_addr":"172.21.15.160:6817/2985674533","cluster_addr":"172.21.15.160:6819/2985674533","heartbeat_back_addr":"172.21.15.160:6823/2985674533","heartbeat_front_addr":"172.21.15.160:6821/2985674533","state":["exists","up"]},{"osd":1,"uuid":"f487a8e6-d5e0-437b-b6ac-145ca95125b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6801","nonce":445311465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6803","nonce":445311465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6807","nonce":445311465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":445311465},{"type":"v1","addr":"172.21.15.160:6805","nonce":445311465}]},"public_addr":"172.21.15.160:6801/445311465","cluster_addr":"172.21.15.160:6803/445311465","heartbeat_back_addr":"172.21.15.160:6807/445311465","heartbeat_front_addr":"172.21.15.160:6805/445311465","state":["exists","up"]},{"osd":2,"uuid":"9269a1cc-1f25-40e9-bdc7-ff9446d26646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6808","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6809","nonce":3852362862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6810","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6811","nonce":3852362862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6814","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6815","nonce":3852362862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6812","nonce":3852362862},{"type":"v1","addr":"172.21.15.160:6813","nonce":3852362862}]},"public_addr":"172.21.15.160:6809/3852362862","cluster_addr":"172.21.15.160:6811/3852362862","heartbeat_back_addr":"172.21.15.160:6815/3852362862","heartbeat_front_addr":"172.21.15.160:6813/3852362862","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.346273+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.287905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:09:28.360519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:09:40.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-16T11:09:40.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-16T11:09:40.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-16T11:09:40.451 INFO:teuthology.orchestra.run.smithi160.stdout:34359738374 2023-12-16T11:09:40.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-16T11:09:40.470 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2023-12-16T11:09:40.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-16T11:09:40.471 INFO:teuthology.orchestra.run.smithi160.stdout:34359738374 2023-12-16T11:09:40.472 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-16T11:09:40.744 INFO:teuthology.orchestra.run.smithi160.stdout:34359738373 2023-12-16T11:09:40.756 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-16T11:09:40.804 INFO:teuthology.orchestra.run.smithi160.stdout:34359738373 2023-12-16T11:09:40.808 INFO:teuthology.orchestra.run.smithi160.stdout:34359738371 2023-12-16T11:09:40.818 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-16T11:09:40.821 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2023-12-16T11:09:41.758 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-16T11:09:41.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-16T11:09:41.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-16T11:09:42.058 INFO:teuthology.orchestra.run.smithi160.stdout:34359738373 2023-12-16T11:09:42.071 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2023-12-16T11:09:42.119 INFO:teuthology.orchestra.run.smithi160.stdout:34359738373 2023-12-16T11:09:42.133 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-16T11:09:42.156 INFO:teuthology.orchestra.run.smithi160.stdout:34359738371 2023-12-16T11:09:42.168 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2023-12-16T11:09:43.073 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-16T11:09:43.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-16T11:09:43.169 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-16T11:09:43.375 INFO:teuthology.orchestra.run.smithi160.stdout:34359738374 2023-12-16T11:09:43.388 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.2 2023-12-16T11:09:43.389 DEBUG:teuthology.parallel:result is None 2023-12-16T11:09:43.430 INFO:teuthology.orchestra.run.smithi160.stdout:34359738374 2023-12-16T11:09:43.443 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2023-12-16T11:09:43.443 DEBUG:teuthology.parallel:result is None 2023-12-16T11:09:43.474 INFO:teuthology.orchestra.run.smithi160.stdout:34359738372 2023-12-16T11:09:43.486 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2023-12-16T11:09:43.486 DEBUG:teuthology.parallel:result is None 2023-12-16T11:09:43.487 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-16T11:09:43.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:09:43.804 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:43.804 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:09:43.817 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-16T11:09:43.683086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82584,"kb_used_data":1516,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032936,"statfs":{"total":289910292480,"available":289825726464,"internally_reserved":0,"allocated":1552384,"data_stored":1284394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001375"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382957+0000","last_change":"2023-12-16T11:09:39.382375+0000","last_active":"2023-12-16T11:09:40.382957+0000","last_peered":"2023-12-16T11:09:40.382957+0000","last_clean":"2023-12-16T11:09:40.382957+0000","last_became_active":"2023-12-16T11:09:39.381887+0000","last_became_peered":"2023-12-16T11:09:39.381887+0000","last_unstale":"2023-12-16T11:09:40.382957+0000","last_undegraded":"2023-12-16T11:09:40.382957+0000","last_fullsized":"2023-12-16T11:09:40.382957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:22:38.622467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382928+0000","last_change":"2023-12-16T11:09:39.380146+0000","last_active":"2023-12-16T11:09:40.382928+0000","last_peered":"2023-12-16T11:09:40.382928+0000","last_clean":"2023-12-16T11:09:40.382928+0000","last_became_active":"2023-12-16T11:09:39.379884+0000","last_became_peered":"2023-12-16T11:09:39.379884+0000","last_unstale":"2023-12-16T11:09:40.382928+0000","last_undegraded":"2023-12-16T11:09:40.382928+0000","last_fullsized":"2023-12-16T11:09:40.382928+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:55:25.028187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382768+0000","last_change":"2023-12-16T11:09:39.382404+0000","last_active":"2023-12-16T11:09:40.382768+0000","last_peered":"2023-12-16T11:09:40.382768+0000","last_clean":"2023-12-16T11:09:40.382768+0000","last_became_active":"2023-12-16T11:09:39.381958+0000","last_became_peered":"2023-12-16T11:09:39.381958+0000","last_unstale":"2023-12-16T11:09:40.382768+0000","last_undegraded":"2023-12-16T11:09:40.382768+0000","last_fullsized":"2023-12-16T11:09:40.382768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T15:45:51.066834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382828+0000","last_change":"2023-12-16T11:09:39.380230+0000","last_active":"2023-12-16T11:09:40.382828+0000","last_peered":"2023-12-16T11:09:40.382828+0000","last_clean":"2023-12-16T11:09:40.382828+0000","last_became_active":"2023-12-16T11:09:39.379849+0000","last_became_peered":"2023-12-16T11:09:39.379849+0000","last_unstale":"2023-12-16T11:09:40.382828+0000","last_undegraded":"2023-12-16T11:09:40.382828+0000","last_fullsized":"2023-12-16T11:09:40.382828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T15:39:47.357099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:41.137026+0000","last_change":"2023-12-16T11:09:39.381744+0000","last_active":"2023-12-16T11:09:41.137026+0000","last_peered":"2023-12-16T11:09:41.137026+0000","last_clean":"2023-12-16T11:09:41.137026+0000","last_became_active":"2023-12-16T11:09:39.381438+0000","last_became_peered":"2023-12-16T11:09:39.381438+0000","last_unstale":"2023-12-16T11:09:41.137026+0000","last_undegraded":"2023-12-16T11:09:41.137026+0000","last_fullsized":"2023-12-16T11:09:41.137026+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:19:22.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382582+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:09:40.382582+0000","last_peered":"2023-12-16T11:09:40.382582+0000","last_clean":"2023-12-16T11:09:40.382582+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:09:40.382582+0000","last_undegraded":"2023-12-16T11:09:40.382582+0000","last_fullsized":"2023-12-16T11:09:40.382582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382507+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:09:40.382507+0000","last_peered":"2023-12-16T11:09:40.382507+0000","last_clean":"2023-12-16T11:09:40.382507+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:09:40.382507+0000","last_undegraded":"2023-12-16T11:09:40.382507+0000","last_fullsized":"2023-12-16T11:09:40.382507+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382871+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:09:40.382871+0000","last_peered":"2023-12-16T11:09:40.382871+0000","last_clean":"2023-12-16T11:09:40.382871+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:09:40.382871+0000","last_undegraded":"2023-12-16T11:09:40.382871+0000","last_fullsized":"2023-12-16T11:09:40.382871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:39.375109+0000","last_change":"2023-12-16T11:09:33.559309+0000","last_active":"2023-12-16T11:09:39.375109+0000","last_peered":"2023-12-16T11:09:39.375109+0000","last_clean":"2023-12-16T11:09:39.375109+0000","last_became_active":"2023-12-16T11:09:33.558255+0000","last_became_peered":"2023-12-16T11:09:33.558255+0000","last_unstale":"2023-12-16T11:09:39.375109+0000","last_undegraded":"2023-12-16T11:09:39.375109+0000","last_fullsized":"2023-12-16T11:09:39.375109+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:15:57.164387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":712704,"data_stored":624408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:09:43.818 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:09:44.092 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:44.093 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:09:44.104 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-16T11:09:43.683086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82584,"kb_used_data":1516,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032936,"statfs":{"total":289910292480,"available":289825726464,"internally_reserved":0,"allocated":1552384,"data_stored":1284394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001375"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382957+0000","last_change":"2023-12-16T11:09:39.382375+0000","last_active":"2023-12-16T11:09:40.382957+0000","last_peered":"2023-12-16T11:09:40.382957+0000","last_clean":"2023-12-16T11:09:40.382957+0000","last_became_active":"2023-12-16T11:09:39.381887+0000","last_became_peered":"2023-12-16T11:09:39.381887+0000","last_unstale":"2023-12-16T11:09:40.382957+0000","last_undegraded":"2023-12-16T11:09:40.382957+0000","last_fullsized":"2023-12-16T11:09:40.382957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:22:38.622467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382928+0000","last_change":"2023-12-16T11:09:39.380146+0000","last_active":"2023-12-16T11:09:40.382928+0000","last_peered":"2023-12-16T11:09:40.382928+0000","last_clean":"2023-12-16T11:09:40.382928+0000","last_became_active":"2023-12-16T11:09:39.379884+0000","last_became_peered":"2023-12-16T11:09:39.379884+0000","last_unstale":"2023-12-16T11:09:40.382928+0000","last_undegraded":"2023-12-16T11:09:40.382928+0000","last_fullsized":"2023-12-16T11:09:40.382928+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:55:25.028187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382768+0000","last_change":"2023-12-16T11:09:39.382404+0000","last_active":"2023-12-16T11:09:40.382768+0000","last_peered":"2023-12-16T11:09:40.382768+0000","last_clean":"2023-12-16T11:09:40.382768+0000","last_became_active":"2023-12-16T11:09:39.381958+0000","last_became_peered":"2023-12-16T11:09:39.381958+0000","last_unstale":"2023-12-16T11:09:40.382768+0000","last_undegraded":"2023-12-16T11:09:40.382768+0000","last_fullsized":"2023-12-16T11:09:40.382768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T15:45:51.066834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382828+0000","last_change":"2023-12-16T11:09:39.380230+0000","last_active":"2023-12-16T11:09:40.382828+0000","last_peered":"2023-12-16T11:09:40.382828+0000","last_clean":"2023-12-16T11:09:40.382828+0000","last_became_active":"2023-12-16T11:09:39.379849+0000","last_became_peered":"2023-12-16T11:09:39.379849+0000","last_unstale":"2023-12-16T11:09:40.382828+0000","last_undegraded":"2023-12-16T11:09:40.382828+0000","last_fullsized":"2023-12-16T11:09:40.382828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T15:39:47.357099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:41.137026+0000","last_change":"2023-12-16T11:09:39.381744+0000","last_active":"2023-12-16T11:09:41.137026+0000","last_peered":"2023-12-16T11:09:41.137026+0000","last_clean":"2023-12-16T11:09:41.137026+0000","last_became_active":"2023-12-16T11:09:39.381438+0000","last_became_peered":"2023-12-16T11:09:39.381438+0000","last_unstale":"2023-12-16T11:09:41.137026+0000","last_undegraded":"2023-12-16T11:09:41.137026+0000","last_fullsized":"2023-12-16T11:09:41.137026+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:19:22.118813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382582+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:09:40.382582+0000","last_peered":"2023-12-16T11:09:40.382582+0000","last_clean":"2023-12-16T11:09:40.382582+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:09:40.382582+0000","last_undegraded":"2023-12-16T11:09:40.382582+0000","last_fullsized":"2023-12-16T11:09:40.382582+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382507+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:09:40.382507+0000","last_peered":"2023-12-16T11:09:40.382507+0000","last_clean":"2023-12-16T11:09:40.382507+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:09:40.382507+0000","last_undegraded":"2023-12-16T11:09:40.382507+0000","last_fullsized":"2023-12-16T11:09:40.382507+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-16T11:09:40.382871+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:09:40.382871+0000","last_peered":"2023-12-16T11:09:40.382871+0000","last_clean":"2023-12-16T11:09:40.382871+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:09:40.382871+0000","last_undegraded":"2023-12-16T11:09:40.382871+0000","last_fullsized":"2023-12-16T11:09:40.382871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-16T11:09:39.375109+0000","last_change":"2023-12-16T11:09:33.559309+0000","last_active":"2023-12-16T11:09:39.375109+0000","last_peered":"2023-12-16T11:09:39.375109+0000","last_clean":"2023-12-16T11:09:39.375109+0000","last_became_active":"2023-12-16T11:09:33.558255+0000","last_became_peered":"2023-12-16T11:09:33.558255+0000","last_unstale":"2023-12-16T11:09:39.375109+0000","last_undegraded":"2023-12-16T11:09:39.375109+0000","last_fullsized":"2023-12-16T11:09:39.375109+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:15:57.164387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":716800,"data_stored":625177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":712704,"data_stored":624408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:09:44.105 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-16T11:09:44.105 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-16T11:09:44.106 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-16T11:09:44.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-16T11:09:44.416 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:09:44.416 INFO:teuthology.orchestra.run.smithi160.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-16T11:09:44.428 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-16T11:09:44.429 INFO:teuthology.run_tasks:Running task exec... 2023-12-16T11:09:44.439 INFO:teuthology.task.exec:Executing custom commands... 2023-12-16T11:09:44.439 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:09:44.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:44.796 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror] 2023-12-16T11:09:44.796 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQAwhX1lzf9TChAAGVGtbmKJfWvxN3ZD/jpM3Q== 2023-12-16T11:09:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-16T11:09:44.797 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:44.797 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror 2023-12-16T11:09:44.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:45.164 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.0] 2023-12-16T11:09:45.164 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQAwhX1lf2nWDRAAxkVkyfgyQar2PwwNxT0taw== 2023-12-16T11:09:45.164 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:45.164 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:45.165 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.0 2023-12-16T11:09:45.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:45.537 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.1] 2023-12-16T11:09:45.537 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQAwhX1lWv21ExAAx1ncs5LGBW8v3Z2r7T0S2Q== 2023-12-16T11:09:45.537 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:45.537 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:45.538 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.1 2023-12-16T11:09:45.550 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:45.903 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.2] 2023-12-16T11:09:45.903 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQAwhX1lwXaHGRAAwpRP9QyVpj0FQYcUWL78uA== 2023-12-16T11:09:45.903 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:45.903 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.2 2023-12-16T11:09:45.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:46.273 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.3] 2023-12-16T11:09:46.273 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQAwhX1lSOlMHxAAAaJDUOPbbj0+tpWi5J1S4g== 2023-12-16T11:09:46.273 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:46.273 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.3 2023-12-16T11:09:46.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:46.640 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror] 2023-12-16T11:09:46.640 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lHI+wIxAA19zSds/Xpr0a3W43OIjwPw== 2023-12-16T11:09:46.640 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-16T11:09:46.640 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:46.640 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror 2023-12-16T11:09:46.653 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:47.009 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.0] 2023-12-16T11:09:47.009 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lesZ7KRAA6LAFqZCdWLTTgTnJwqfiiQ== 2023-12-16T11:09:47.010 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:47.010 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:47.010 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.0 2023-12-16T11:09:47.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:47.380 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.1] 2023-12-16T11:09:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lGYpILxAAUyQAZnFZsnlD3A4/g62wVw== 2023-12-16T11:09:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:47.380 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:47.380 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.1 2023-12-16T11:09:47.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:47.746 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.2] 2023-12-16T11:09:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1lbegpNRAAFsQDUF2RrMCcOYzwRacPeQ== 2023-12-16T11:09:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:47.746 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.2 2023-12-16T11:09:47.759 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-16T11:09:48.110 INFO:teuthology.orchestra.run.smithi160.stdout:[client.mirror.3] 2023-12-16T11:09:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQBUhX1l1hz8OhAAdTv2CENTOBvJrAKiT0FCBA== 2023-12-16T11:09:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: caps mon = "profile rbd-mirror" 2023-12-16T11:09:48.110 INFO:teuthology.orchestra.run.smithi160.stdout: caps osd = "profile rbd" 2023-12-16T11:09:48.111 INFO:teuthology.orchestra.run.smithi160.stderr:updated caps for client.mirror.3 2023-12-16T11:09:48.123 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-16T11:09:48.136 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-16T11:09:48.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-16T11:09:48.138 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:09:48.179 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:09:48.181 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:09:48.181 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-16T11:09:48.213 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-16T11:09:48.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-16T11:09:48.215 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-16T11:09:48.247 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-16T11:09:48.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-16T11:09:48.249 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-16T11:09:48.282 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-16T11:09:48.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-16T11:09:48.286 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-16T11:09:48.317 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-16T11:09:48.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-16T11:09:48.320 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-16T11:09:48.348 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-16T11:09:48.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-16T11:09:48.350 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-16T11:09:48.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-16T11:09:48.382 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-16T11:09:48.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-16T11:09:48.385 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-16T11:09:48.417 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702724988 2023-12-16T11:09:48.417 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-16T11:09:48.417 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-16T11:09:48.447 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702724988 2023-12-16T11:09:48.447 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-16T11:09:48.447 INFO:teuthology.run_tasks:Running task workunit... 2023-12-16T11:09:48.485 INFO:tasks.workunit:Pulling workunits from ref 6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T11:09:48.485 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-16T11:09:48.485 DEBUG:teuthology.orchestra.run.smithi160:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-16T11:09:48.486 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-16T11:09:48.486 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-16T11:09:48.486 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-16T11:09:48.486 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-12-16T11:09:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T11:09:48.501 INFO:teuthology.orchestra.run.smithi160.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-16T11:09:48.501 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-16T11:09:48.556 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-16T11:09:48.556 DEBUG:teuthology.orchestra.run.smithi160:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-16T11:09:48.611 INFO:tasks.workunit:timeout=6h 2023-12-16T11:09:48.611 INFO:tasks.workunit:cleanup=True 2023-12-16T11:09:48.611 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T11:09:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-16T11:10:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Note: switching to '6d933094a156ec7587742e5aa25fba5ba27ec678'. 2023-12-16T11:10:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:10:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-16T11:10:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-16T11:10:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:state without impacting any branches by switching back to a branch. 2023-12-16T11:10:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: git switch -c 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Or undo this operation with: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: git switch - 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:10:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:HEAD is now at 6d933094a15 Merge PR #52196 into main 2023-12-16T11:10:34.142 DEBUG:teuthology.orchestra.run.smithi160:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-16T11:10:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-16T11:10:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-16T11:10:34.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-16T11:10:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-16T11:10:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-16T11:10:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-16T11:10:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-16T11:10:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-16T11:10:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-16T11:10:35.155 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-12-16T11:10:35.156 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-16T11:10:35.211 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-16T11:10:35.212 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-16T11:10:35.213 DEBUG:teuthology.orchestra.run.smithi160:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6d933094a156ec7587742e5aa25fba5ba27ec678 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-16T11:10:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ IMAGE_COUNT=50 2023-12-16T11:10:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ export LOCKDEP=0 2023-12-16T11:10:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ LOCKDEP=0 2023-12-16T11:10:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ type xmlstarlet 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ XMLSTARLET=xmlstarlet 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ RBD_MIRROR_INSTANCES=4 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ CLUSTER1=cluster1 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ CLUSTER2=cluster2 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ PEER_CLUSTER_SUFFIX= 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ POOL=mirror 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ PARENT_POOL=mirror_parent 2023-12-16T11:10:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ NS1=ns1 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ NS2=ns2 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ TEMPDIR= 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ CEPH_ID=mirror 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ RBD_IMAGE_FEATURES=layering 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ RBD_MIRROR_MODE=snapshot 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ MIRROR_POOL_MODE=image 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' snapshot = snapshot ']' 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ MIRROR_POOL_MODE=image 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ CEPH_ARGS='--id mirror' 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ LAST_MIRROR_INSTANCE=3 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-16T11:10:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-16T11:10:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-16T11:10:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ CEPH_BIN=. 2023-12-16T11:10:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ CEPH_SRC=. 2023-12-16T11:10:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' -e CMakeCache.txt ']' 2023-12-16T11:10:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 -gt 0 ']' 2023-12-16T11:10:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ setup 2023-12-16T11:10:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local c 2023-12-16T11:10:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-12-16T11:10:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ setup_tempdir 2023-12-16T11:10:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mktemp -d 2023-12-16T11:10:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ TEMPDIR=/tmp/tmp.jvOAE3Umds 2023-12-16T11:10:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -z 1 ']' 2023-12-16T11:10:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ setup_pools cluster1 cluster2 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local remote_cluster=cluster2 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mon_map_file 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mon_addr 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local admin_key_file 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local uuid 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-16T11:10:36.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror' already exists 2023-12-16T11:10:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-16T11:10:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror_parent' already exists 2023-12-16T11:10:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 pool init mirror 2023-12-16T11:10:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-12-16T11:10:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-16T11:10:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-16T11:10:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-16T11:10:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-16T11:10:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-16T11:10:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -z ']' 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -z ']' 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_add cluster1 mirror cluster2 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local client_cluster=cluster2 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local remote_cluster=cluster2 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local uuid_var_name 2023-12-16T11:10:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local error_code 2023-12-16T11:10:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local peer_uuid 2023-12-16T11:10:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 16 32 2023-12-16T11:10:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:10:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-16T11:10:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_uuid=76afda8f-dbc9-4634-9c43-a56db8f5014b 2023-12-16T11:10:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:10:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 17 ']' 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 -eq 0 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror_parent 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local client_cluster=cluster2 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local remote_cluster=cluster2 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local uuid_var_name 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local error_code 2023-12-16T11:10:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local peer_uuid 2023-12-16T11:10:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 16 32 2023-12-16T11:10:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:10:43.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_uuid=f6414695-54b1-4636-bc8c-fad1ea4f1bcd 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 17 ']' 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 -eq 0 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ setup_pools cluster2 cluster1 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:10:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local remote_cluster=cluster1 2023-12-16T11:10:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mon_map_file 2023-12-16T11:10:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mon_addr 2023-12-16T11:10:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local admin_key_file 2023-12-16T11:10:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local uuid 2023-12-16T11:10:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-16T11:10:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror' already exists 2023-12-16T11:10:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:44.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-16T11:10:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror_parent' already exists 2023-12-16T11:10:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:45.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 pool init mirror 2023-12-16T11:10:48.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:10:48.275+0000 7ff740c21640 -1 rbd::mirror::PoolReplayer: 0x55c1dcf16c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:48.305 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:10:48.305+0000 7f4553956640 -1 rbd::mirror::PoolReplayer: 0x560d103d4c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:48.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:10:48.331+0000 7f962f011640 -1 rbd::mirror::PoolReplayer: 0x563ab62f4c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:48.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:10:48.364+0000 7f4553956640 -1 rbd::mirror::PoolReplayer: 0x560d10555500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:48.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:10:48.370+0000 7ff4cd223640 -1 rbd::mirror::PoolReplayer: 0x558f3402ac00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:48.370 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:10:48.370+0000 7f962f011640 -1 rbd::mirror::PoolReplayer: 0x563ab6485500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-12-16T11:10:48.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:10:48.419+0000 7ff4cd223640 -1 rbd::mirror::PoolReplayer: 0x558f341ef500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:48.514 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:10:48.514+0000 7ff740c21640 -1 rbd::mirror::PoolReplayer: 0x55c1dd08d500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-16T11:10:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:10:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-16T11:10:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-16T11:10:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-16T11:10:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-16T11:10:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-16T11:10:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -z ']' 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -z ']' 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_add cluster2 mirror cluster1 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local client_cluster=cluster1 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local remote_cluster=cluster1 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local uuid_var_name 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local error_code 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local peer_uuid 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 16 32 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:10:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_uuid=264f766c-64a2-4932-922a-147bb4d9d99e 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 17 ']' 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 -eq 0 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror_parent 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local client_cluster=cluster1 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local remote_cluster=cluster1 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local uuid_var_name 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local error_code 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local peer_uuid 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 16 32 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-16T11:10:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ peer_uuid=e8948614-e206-4940-8569-0a8d3c65b1e1 2023-12-16T11:10:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:10:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:10:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 17 ']' 2023-12-16T11:10:51.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 -eq 0 2023-12-16T11:10:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:10:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:10:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirrors cluster1 2023-12-16T11:10:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:10:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 0 3 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster1:0 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:0 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster1:0 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster1 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=0 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 = cluster1:0 ']' 2023-12-16T11:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster1 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=0 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster1:1 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:1 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster1:1 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster1 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=1 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 1 = cluster1:1 ']' 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster1 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=1 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster1:2 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:2 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster1:2 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster1 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=2 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 2 = cluster1:2 ']' 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster1 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=2 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster1:3 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:3 2023-12-16T11:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster1:3 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster1 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=3 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 3 = cluster1:3 ']' 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster1 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=3 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirrors cluster2 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:10:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 0 3 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster2:0 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:0 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster2:0 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster2 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=0 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 = cluster2:0 ']' 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster2 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster2 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=0 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:10:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster2:1 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:1 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster2:1 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster2 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=1 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 1 = cluster2:1 ']' 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster2 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster2 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=1 2023-12-16T11:10:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster2:2 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:2 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster2:2 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster2 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=2 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 2 = cluster2:2 ']' 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster2 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster2 2023-12-16T11:10:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=2 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster2:3 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:3 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster2:3 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster2 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=3 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 3 = cluster2:3 ']' 2023-12-16T11:10:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster2 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster2 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=3 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee -a /tmp/tmp.jvOAE3Umds/rbd-mirror.test.log 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ date '+%F %T' 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo 2023-12-16 11:10:51 TEST: add image and test replay after client crashes 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:2023-12-16 11:10:51 TEST: add image and test replay after client crashes 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=test 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror test 512M 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:10:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:10:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:10:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2023-12-16T11:10:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:10:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:10:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:10:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-16T11:10:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:10:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:10:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:10:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-16T11:10:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:10:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-16T11:10:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10bf6626b7ea 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10bf6626b7ea 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:10:51 2023 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:10:51 2023 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:10:51 2023 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:10:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:10:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:10:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:10:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:10:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:10:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:10:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:10:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:10:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:10:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:10:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:10:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:10:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:10:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:10:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:10:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:10:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:10:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:10:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:10:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:10:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:10:53.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:10:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:10:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:10:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:10:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:10:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:10:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:10:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:53.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:10:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:53.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:10:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 1 2023-12-16T11:10:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:10:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:10:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:10:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:10:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:10:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:10:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:10:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:10:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:10:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:10:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:10:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:10:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:10:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:10:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:10:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:10:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:10:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:10:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:55.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:10:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:10:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:10:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:10:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:10:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:10:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:10:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:10:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:10:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:10:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:10:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:10:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 2 2023-12-16T11:10:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:10:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:10:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:10:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:10:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:10:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:10:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:10:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:10:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:10:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:10:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:10:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:10:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:10:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:10:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:10:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:10:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:10:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:10:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:10:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:10:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:10:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:10:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:10:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:10:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:10:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:10:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:10:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:10:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:10:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:10:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:10:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:10:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:10:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:10:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:10:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:10:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:10:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:10:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:10:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:10:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:10:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:10:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:10:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:10:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:10:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:10:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:10:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:10:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:10:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:10:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:10:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:10:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 4 2023-12-16T11:10:58.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2632840123750433) > weight (0.25) 2023-12-16T11:10:58.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9687582287968278) > weight (0.25) 2023-12-16T11:10:58.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3050929323770156) > weight (0.25) 2023-12-16T11:10:58.488 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5350832695740643) > weight (0.25) 2023-12-16T11:10:58.488 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-16T11:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:11:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:11:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:11:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:11:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:11:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:11:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:11:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:11:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:11:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:11:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:11:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:11:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:01.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:01.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:11:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:11:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:11:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:11:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:11:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:11:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:11:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:11:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:11:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:11:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:11:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:11:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:11:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:11:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:11:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:11:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:11:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 8 2023-12-16T11:11:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:11:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:11:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:11:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:11:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:11:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:11:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:11:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:11:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:10.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:11:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:11:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:11:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:11:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:10.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:11:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:11:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:11:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:11:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:11:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:11:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:11:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:11:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:11:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:11:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:11:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:11:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:11:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 8 2023-12-16T11:11:18.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:11:18.373+0000 7f4553956640 -1 rbd::mirror::Mirror: 0x560d0f59dd40 update_pool_replayers: restarting failed pool replayer for uuid: 76afda8f-dbc9-4634-9c43-a56db8f5014b cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.378 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:11:18.378+0000 7f962f011640 -1 rbd::mirror::Mirror: 0x563ab54cdd40 update_pool_replayers: restarting failed pool replayer for uuid: 76afda8f-dbc9-4634-9c43-a56db8f5014b cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:11:18.425+0000 7f4553956640 -1 rbd::mirror::Mirror: 0x560d0f59dd40 update_pool_replayers: restarting failed pool replayer for uuid: f6414695-54b1-4636-bc8c-fad1ea4f1bcd cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:11:18.427+0000 7ff4cd223640 -1 rbd::mirror::Mirror: 0x558f33237d40 update_pool_replayers: restarting failed pool replayer for uuid: 76afda8f-dbc9-4634-9c43-a56db8f5014b cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:11:18.440+0000 7f962f011640 -1 rbd::mirror::Mirror: 0x563ab54cdd40 update_pool_replayers: restarting failed pool replayer for uuid: f6414695-54b1-4636-bc8c-fad1ea4f1bcd cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:11:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:11:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:11:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:11:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:11:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:11:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:11:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:11:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:11:18.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:11:18.525+0000 7ff740c21640 -1 rbd::mirror::Mirror: 0x55c1dc0dfd40 update_pool_replayers: restarting failed pool replayer for uuid: 76afda8f-dbc9-4634-9c43-a56db8f5014b cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:11:18.545+0000 7ff4cd223640 -1 rbd::mirror::Mirror: 0x558f33237d40 update_pool_replayers: restarting failed pool replayer for uuid: f6414695-54b1-4636-bc8c-fad1ea4f1bcd cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:18.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:11:18.609+0000 7ff740c21640 -1 rbd::mirror::Mirror: 0x55c1dc0dfd40 update_pool_replayers: restarting failed pool replayer for uuid: f6414695-54b1-4636-bc8c-fad1ea4f1bcd cluster: cluster2 client: client.mirror 2023-12-16T11:11:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:11:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:11:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:11:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:11:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:11:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:11:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:11:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:11:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:11:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:11:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:11:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:11:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:11:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:11:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:11:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:11:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 8 2023-12-16T11:11:26.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9832801684125874) > weight (0.25) 2023-12-16T11:11:26.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7451036688202668) > weight (0.25) 2023-12-16T11:11:26.487 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.41698751485171504) > weight (0.25) 2023-12-16T11:11:26.488 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.927570938567679) > weight (0.25) 2023-12-16T11:11:26.488 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2023-12-16T11:11:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:11:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:11:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:11:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:11:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:11:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:11:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:11:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:11:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:11:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:11:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:11:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:11:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:11:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:11:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:11:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:11:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:11:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:11:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:11:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:11:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:11:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:11:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:11:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:11:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:11:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:11:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:11:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name= 2023-12-16T11:11:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 10 2023-12-16T11:11:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:11:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:11:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:11:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:11:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=11 2023-12-16T11:11:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:11:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:11:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 85331968 bytes 2023-12-16T11:11:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 124694528 bytes 2023-12-16T11:11:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 216819712 bytes 2023-12-16T11:11:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 256098304 bytes 2023-12-16T11:11:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 263869440 bytes 2023-12-16T11:11:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 328076288 bytes 2023-12-16T11:11:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 388486144 bytes 2023-12-16T11:11:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 471489536 bytes 2023-12-16T11:11:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 528973824 bytes 2023-12-16T11:11:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 602496000 bytes 2023-12-16T11:11:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 618404864 bytes 2023-12-16T11:11:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 619027456 bytes 2023-12-16T11:11:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 647396352 bytes 2023-12-16T11:11:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 675939328 bytes 2023-12-16T11:11:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 703983616 bytes 2023-12-16T11:11:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 798430208 bytes 2023-12-16T11:11:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 880285696 bytes 2023-12-16T11:11:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 899306496 bytes 2023-12-16T11:11:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 919375872 bytes 2023-12-16T11:11:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 982480896 bytes 2023-12-16T11:11:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1013691392 bytes 2023-12-16T11:11:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1026211840 bytes 2023-12-16T11:11:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1044916224 bytes 2023-12-16T11:11:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1134713856 bytes 2023-12-16T11:11:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1161745408 bytes 2023-12-16T11:11:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1213360128 bytes 2023-12-16T11:11:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1277057024 bytes 2023-12-16T11:11:31.578 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1330597888 bytes 2023-12-16T11:11:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1333728256 bytes 2023-12-16T11:11:31.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1383644160 bytes 2023-12-16T11:11:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1407439872 bytes 2023-12-16T11:11:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1431760896 bytes 2023-12-16T11:11:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1474552832 bytes 2023-12-16T11:11:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1499058176 bytes 2023-12-16T11:11:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1590959104 bytes 2023-12-16T11:11:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1633885184 bytes 2023-12-16T11:11:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1634502656 bytes 2023-12-16T11:11:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1669634048 bytes 2023-12-16T11:11:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1710913536 bytes 2023-12-16T11:11:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1775506432 bytes 2023-12-16T11:11:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1817526272 bytes 2023-12-16T11:11:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1876263936 bytes 2023-12-16T11:11:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1890212864 bytes 2023-12-16T11:11:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1969148928 bytes 2023-12-16T11:11:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2042245120 bytes 2023-12-16T11:11:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2080279552 bytes 2023-12-16T11:11:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2136577024 bytes 2023-12-16T11:11:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2257858560 bytes 2023-12-16T11:11:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2302641152 bytes 2023-12-16T11:11:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2333633536 bytes 2023-12-16T11:11:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2460484608 bytes 2023-12-16T11:11:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2535116800 bytes 2023-12-16T11:11:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2574015488 bytes 2023-12-16T11:11:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2594049024 bytes 2023-12-16T11:11:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2627944448 bytes 2023-12-16T11:11:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2631555072 bytes 2023-12-16T11:11:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2691408896 bytes 2023-12-16T11:11:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2730570752 bytes 2023-12-16T11:11:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2784441344 bytes 2023-12-16T11:11:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2850181120 bytes 2023-12-16T11:11:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2897833984 bytes 2023-12-16T11:11:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2898422784 bytes 2023-12-16T11:11:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2937932800 bytes 2023-12-16T11:11:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2986312704 bytes 2023-12-16T11:11:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 3022390272 bytes 2023-12-16T11:11:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 3047744512 bytes 2023-12-16T11:11:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 3107409920 bytes 2023-12-16T11:11:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 3162624000 bytes 2023-12-16T11:11:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 3231493120 bytes 2023-12-16T11:11:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 3239993344 bytes 2023-12-16T11:11:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 3253137408 bytes 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:11:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:11:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:11:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:11:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:11:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:11:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:11:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:11:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:11:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:11:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:11:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:11:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:11:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:11:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:11:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:11:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:11:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:11:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:11:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:11:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:11:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:11:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:11:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:11:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:11:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:11:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:11:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:11:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:11:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:11:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:11:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:11:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:11:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:11:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:11:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:11:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:11:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:11:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:11:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:11:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:11:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:11:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:11:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:11:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:11:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:11:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:11:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:11:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:11:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:11:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702725052,"remote_snapshot_timestamp":1702725052,"replay_state":"idle"} 2023-12-16T11:11:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.1 on smithi160 2023-12-16T11:11:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:11:48 2023-12-16T11:11:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:11:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:11:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:11:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:11:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:11:48 2023-12-16T11:11:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:11:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702725052,"remote_snapshot_timestamp":1702725052,"replay_state":"idle"} 2023-12-16T11:11:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:11:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:11:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:11:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:11:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.1 on smithi160 2023-12-16T11:11:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:11:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:11:48' 2023-12-16T11:11:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:11:48' 2023-12-16T11:11:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:11:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:11:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:11:48' 2023-12-16T11:11:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:11:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:11:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:11:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:11:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:11:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:11:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:11:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:11:48' 2023-12-16T11:11:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:11:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:11:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:11:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:11:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap1 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap1 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap1 2023-12-16T11:11:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:12:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:12:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:12:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:12:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:12:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:12:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:12:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:12:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:12:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:12:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:12:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:12:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:12:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:12:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:12:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:12:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:12:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:12:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:12:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:12:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:12:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:12:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:12:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:12:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:12:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:12:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:12:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:12:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:12:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:12:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:12:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:12:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:12:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:12:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:12:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:12:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:12:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:12:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:12:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:12:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:12:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:12:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:12:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:12:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:12:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:12:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:12:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:12:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:12:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 6 2023-12-16T11:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:12:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:12:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=6 2023-12-16T11:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:12:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:12:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:12:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:12:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:12:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=4 2023-12-16T11:12:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 6 = 4 2023-12-16T11:12:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:12:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:12:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:12:11.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:12:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:12:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:12:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:12:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:12:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:12:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:12:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 6 = '' 2023-12-16T11:12:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:12:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:12:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:12:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:12:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:12:12.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:12:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:12:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:12:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:12:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:12:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 6 = '' 2023-12-16T11:12:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:12:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:12:13.489 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6628780250492968) > weight (0.25) 2023-12-16T11:12:13.489 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-16T11:12:13.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-16T11:12:13.490 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-16T11:12:13.490 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:12:13.490+0000 7f45508fa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 91511) UID: 1000 2023-12-16T11:12:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:12:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:12:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=6 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 6 = 6 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap1 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:12:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:12:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-16T11:12:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:12:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:12:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:12:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:12:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:12:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:12:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap1 2023-12-16T11:12:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap1 2023-12-16T11:12:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:12:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:12:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:12:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:12:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:12:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:12:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:12:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:12:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:12:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:12:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:12:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap1-clean 2023-12-16T11:12:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-16T11:12:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:12:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap1-clean 2023-12-16T11:12:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:12:16.490 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-16T11:12:16.491 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:12:16.491 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-16T11:12:16.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:12:16.491 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:12:16.494 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:12:16.494 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-16T11:12:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:12:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:12:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:12:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:12:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=13 2023-12-16T11:12:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:12:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:12:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 26663936 bytes 2023-12-16T11:12:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 111343616 bytes 2023-12-16T11:12:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 148845568 bytes 2023-12-16T11:12:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 230272000 bytes 2023-12-16T11:12:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 261099520 bytes 2023-12-16T11:12:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 317668352 bytes 2023-12-16T11:12:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 349257728 bytes 2023-12-16T11:12:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 425736192 bytes 2023-12-16T11:12:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 512889856 bytes 2023-12-16T11:12:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 528592896 bytes 2023-12-16T11:12:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 559659008 bytes 2023-12-16T11:12:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 610394112 bytes 2023-12-16T11:12:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 619996160 bytes 2023-12-16T11:12:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 667695104 bytes 2023-12-16T11:12:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 702415872 bytes 2023-12-16T11:12:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 746811392 bytes 2023-12-16T11:12:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 804756480 bytes 2023-12-16T11:12:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 854291456 bytes 2023-12-16T11:12:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 914822144 bytes 2023-12-16T11:12:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 970890240 bytes 2023-12-16T11:12:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 997820416 bytes 2023-12-16T11:12:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1026440192 bytes 2023-12-16T11:12:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1048312832 bytes 2023-12-16T11:12:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1076146176 bytes 2023-12-16T11:12:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1131528192 bytes 2023-12-16T11:12:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1136283648 bytes 2023-12-16T11:12:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1182551040 bytes 2023-12-16T11:12:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1236667392 bytes 2023-12-16T11:12:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1293539328 bytes 2023-12-16T11:12:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1342043136 bytes 2023-12-16T11:12:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1342653440 bytes 2023-12-16T11:12:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1389093888 bytes 2023-12-16T11:12:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1419091968 bytes 2023-12-16T11:12:22.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1468171264 bytes 2023-12-16T11:12:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1544007680 bytes 2023-12-16T11:12:22.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1596762112 bytes 2023-12-16T11:12:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1601497088 bytes 2023-12-16T11:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1605637120 bytes 2023-12-16T11:12:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1666864128 bytes 2023-12-16T11:12:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1689416704 bytes 2023-12-16T11:12:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1727621120 bytes 2023-12-16T11:12:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1741681664 bytes 2023-12-16T11:12:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1783495680 bytes 2023-12-16T11:12:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1812490240 bytes 2023-12-16T11:12:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1868485632 bytes 2023-12-16T11:12:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1869804544 bytes 2023-12-16T11:12:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1985587200 bytes 2023-12-16T11:12:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2015607808 bytes 2023-12-16T11:12:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2030225408 bytes 2023-12-16T11:12:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2076798976 bytes 2023-12-16T11:12:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2080959488 bytes 2023-12-16T11:12:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2133264384 bytes 2023-12-16T11:12:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2177859584 bytes 2023-12-16T11:12:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2217182208 bytes 2023-12-16T11:12:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2220650496 bytes 2023-12-16T11:12:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2259714048 bytes 2023-12-16T11:12:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2278282240 bytes 2023-12-16T11:12:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2314052608 bytes 2023-12-16T11:12:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2339135488 bytes 2023-12-16T11:12:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2389635072 bytes 2023-12-16T11:12:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2430205952 bytes 2023-12-16T11:12:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2494359552 bytes 2023-12-16T11:12:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2533765120 bytes 2023-12-16T11:12:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2566654976 bytes 2023-12-16T11:12:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2608609280 bytes 2023-12-16T11:12:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2619325440 bytes 2023-12-16T11:12:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2649480192 bytes 2023-12-16T11:12:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2764802048 bytes 2023-12-16T11:12:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2789169152 bytes 2023-12-16T11:12:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2805091328 bytes 2023-12-16T11:12:27.489 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29034243683906014) > weight (0.25) 2023-12-16T11:12:27.489 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-16T11:12:27.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-16T11:12:27.489 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-16T11:12:27.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:12:27.489+0000 7f3041be8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 91670) UID: 1000 2023-12-16T11:12:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 2843061248 bytes 2023-12-16T11:12:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 2890482688 bytes 2023-12-16T11:12:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 2926432256 bytes 2023-12-16T11:12:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 3009408000 bytes 2023-12-16T11:12:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 3052863488 bytes 2023-12-16T11:12:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 3111519232 bytes 2023-12-16T11:12:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3157534720 bytes 2023-12-16T11:12:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3183245312 bytes 2023-12-16T11:12:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3235407872 bytes 2023-12-16T11:12:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8000 IOs, 3293133824 bytes 2023-12-16T11:12:29.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8100 IOs, 3303530496 bytes 2023-12-16T11:12:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8200 IOs, 3381804032 bytes 2023-12-16T11:12:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8300 IOs, 3398732800 bytes 2023-12-16T11:12:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8400 IOs, 3439769600 bytes 2023-12-16T11:12:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8500 IOs, 3523499008 bytes 2023-12-16T11:12:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8600 IOs, 3529375744 bytes 2023-12-16T11:12:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8700 IOs, 3617126400 bytes 2023-12-16T11:12:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8800 IOs, 3664800768 bytes 2023-12-16T11:12:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8900 IOs, 3712018432 bytes 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:30.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:12:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:12:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:12:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:12:30.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:12:30.490 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-16T11:12:30.490 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:12:30.490 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-16T11:12:30.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-16T11:12:30.491 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-16T11:12:30.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-16T11:12:30.494 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-16T11:12:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:12:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:12:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:12:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:12:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:12:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:12:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:12:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:12:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:12:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:12:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:12:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:12:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:12:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:12:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:12:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:12:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:12:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:12:18 2023-12-16T11:12:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:12:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:12:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:12:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:12:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:12:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:12:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:12:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:12:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:12:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:12:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:12:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:12:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:12:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:12:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:12:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:12:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:12:18 2023-12-16T11:12:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:12:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:12:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:12:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:12:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:12:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:12:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:12:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:12:18 2023-12-16T11:12:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:12:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:12:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:12:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:12:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:12:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:12:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:12:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:12:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:12:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:12:18 2023-12-16T11:12:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:12:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:12:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:12:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:12:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:12:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:12:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:12:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:12:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:12:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:12:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:12:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:12:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702725130,"remote_snapshot_timestamp":1702725130,"replay_state":"idle"} 2023-12-16T11:12:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.2 on smithi160 2023-12-16T11:12:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:12:48 2023-12-16T11:12:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:12:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:12:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:12:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:12:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:12:48 2023-12-16T11:12:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:12:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702725130,"remote_snapshot_timestamp":1702725130,"replay_state":"idle"} 2023-12-16T11:12:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:12:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:12:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:12:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:12:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.2 on smithi160 2023-12-16T11:12:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:12:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:12:48' 2023-12-16T11:12:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:12:48' 2023-12-16T11:12:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:12:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:12:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:12:48' 2023-12-16T11:12:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:12:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:12:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:12:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:12:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:12:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:12:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:12:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:12:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:12:48' 2023-12-16T11:12:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:12:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:12:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:12:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:12:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:12:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:12:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:12:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap2 2023-12-16T11:12:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap2 2023-12-16T11:12:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:12:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:12:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:12:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap2 2023-12-16T11:12:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:13:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:13:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:13:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:13:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:13:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:13:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:13:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:13:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:13:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:13:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:13:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:13:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:13:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:13:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:13:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:13:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:13:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:13:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:13:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:13:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:13:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:13:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:13:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:13:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:13:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:13:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:13:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:13:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:13:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:13:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:13:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:13:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:13:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:13:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:13:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:13:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:13:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:13:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:13:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:13:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:13:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:13:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:13:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:13:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:13:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:13:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:13:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:13:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:13:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:13:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:13:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:13:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:13:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:13:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:13:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:13:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:13:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:13:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:13:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:13:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:13:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:13:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:13:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:13:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 9 2023-12-16T11:13:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:13:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:13:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:13:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:13:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=9 2023-12-16T11:13:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:03.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:13:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:13:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:13:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:13:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:13:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 9 = '' 2023-12-16T11:13:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:13:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:13:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:13:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:13:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 9 = '' 2023-12-16T11:13:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:05.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:13:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:13:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:13:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:13:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 9 = '' 2023-12-16T11:13:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:13:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=9 2023-12-16T11:13:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 9 = 9 2023-12-16T11:13:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-16T11:13:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap2 2023-12-16T11:13:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:13:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:13:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:13:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 4 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:13:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:13:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:13:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:13:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:13:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:13:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:13:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:13:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:13:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap1-clean ']' 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap1-clean 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap2 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap2 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap2-clean 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap2-clean 2023-12-16T11:13:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:13:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:13:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:13:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:13:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=8 2023-12-16T11:13:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:13:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:13:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 33239040 bytes 2023-12-16T11:13:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 54071296 bytes 2023-12-16T11:13:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 102491136 bytes 2023-12-16T11:13:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 124861440 bytes 2023-12-16T11:13:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 167424000 bytes 2023-12-16T11:13:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 234448896 bytes 2023-12-16T11:13:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 260936704 bytes 2023-12-16T11:13:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 261554176 bytes 2023-12-16T11:13:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 290176000 bytes 2023-12-16T11:13:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 315446272 bytes 2023-12-16T11:13:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 336566272 bytes 2023-12-16T11:13:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 360653824 bytes 2023-12-16T11:13:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 400329728 bytes 2023-12-16T11:13:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 426835968 bytes 2023-12-16T11:13:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 434819072 bytes 2023-12-16T11:13:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 508552192 bytes 2023-12-16T11:13:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 541382656 bytes 2023-12-16T11:13:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 590414848 bytes 2023-12-16T11:13:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 677704704 bytes 2023-12-16T11:13:14.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 711181312 bytes 2023-12-16T11:13:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 725762048 bytes 2023-12-16T11:13:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 768374784 bytes 2023-12-16T11:13:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 817244160 bytes 2023-12-16T11:13:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 867492864 bytes 2023-12-16T11:13:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 892331008 bytes 2023-12-16T11:13:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 943337472 bytes 2023-12-16T11:13:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 973053952 bytes 2023-12-16T11:13:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1000557568 bytes 2023-12-16T11:13:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1100298240 bytes 2023-12-16T11:13:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1127605248 bytes 2023-12-16T11:13:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1168398336 bytes 2023-12-16T11:13:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1193655296 bytes 2023-12-16T11:13:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1223808000 bytes 2023-12-16T11:13:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1347535872 bytes 2023-12-16T11:13:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1401523200 bytes 2023-12-16T11:13:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1427918848 bytes 2023-12-16T11:13:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1441241088 bytes 2023-12-16T11:13:16.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1444785152 bytes 2023-12-16T11:13:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1457725440 bytes 2023-12-16T11:13:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1460568064 bytes 2023-12-16T11:13:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1548133376 bytes 2023-12-16T11:13:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1623840768 bytes 2023-12-16T11:13:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1644048384 bytes 2023-12-16T11:13:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1745503232 bytes 2023-12-16T11:13:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1778417664 bytes 2023-12-16T11:13:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1794656256 bytes 2023-12-16T11:13:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1841213440 bytes 2023-12-16T11:13:18.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1851152384 bytes 2023-12-16T11:13:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1897829376 bytes 2023-12-16T11:13:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1918804992 bytes 2023-12-16T11:13:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1939846144 bytes 2023-12-16T11:13:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 1983697920 bytes 2023-12-16T11:13:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2147718144 bytes 2023-12-16T11:13:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2166677504 bytes 2023-12-16T11:13:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2187896832 bytes 2023-12-16T11:13:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2191955968 bytes 2023-12-16T11:13:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:13:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:13:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:13:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:13:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:13:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:13:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:13:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:13:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:13:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:13:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:13:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":17834188.8,"bytes_per_snapshot":535025664.0,"last_snapshot_bytes":535025664,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725182,"remote_snapshot_timestamp":1702725182,"replay_state":"idle"} 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.2 on smithi160 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:13:18 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:13:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:13:18 2023-12-16T11:13:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:13:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":17834188.8,"bytes_per_snapshot":535025664.0,"last_snapshot_bytes":535025664,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725182,"remote_snapshot_timestamp":1702725182,"replay_state":"idle"} 2023-12-16T11:13:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:13:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:13:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:13:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:13:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.2 on smithi160 2023-12-16T11:13:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:13:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:13:18' 2023-12-16T11:13:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:13:18' 2023-12-16T11:13:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:13:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:13:18' 2023-12-16T11:13:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:13:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:13:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:13:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:13:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:13:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:13:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:13:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:13:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:13:18' 2023-12-16T11:13:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:13:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:13:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:13:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:13:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap3 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap3 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap3 2023-12-16T11:13:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:13:26.495 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-16T11:13:26.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-16T11:13:26.496 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-16T11:13:26.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:13:26.496+0000 7ff73d65c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 91493) UID: 1000 2023-12-16T11:13:30.496 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-16T11:13:30.496 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:13:30.497 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-16T11:13:30.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-16T11:13:30.497 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-16T11:13:30.500 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-16T11:13:30.500 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-12-16T11:13:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:13:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:13:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:13:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:13:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:13:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:13:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:13:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:13:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:13:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:13:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:13:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:13:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:13:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:13:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:13:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:13:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:13:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:13:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:13:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:13:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:13:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:13:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:13:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:13:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:13:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:13:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:13:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:13:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:13:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:13:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:13:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:13:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:13:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:13:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:13:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:13:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:13:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:13:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:13:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:13:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:13:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:13:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:13:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:13:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:13:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:13:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:13:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:13:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:13:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:13:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:13:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:13:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:13:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:13:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:13:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:13:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:13:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:13:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:13:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:13:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 13 2023-12-16T11:13:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:13:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:13:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:13:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:13:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=13 2023-12-16T11:13:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:13:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:13:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:13:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:13:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:13:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 13 = '' 2023-12-16T11:13:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:13:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:13:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:13:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:13:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:13:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 13 = '' 2023-12-16T11:13:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:13:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:13:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:13:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:13:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:13:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:13:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 13 = '' 2023-12-16T11:13:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:13:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:13:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:13:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:13:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:13:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:13:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=13 2023-12-16T11:13:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 13 = 13 2023-12-16T11:13:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-16T11:13:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:13:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap3 2023-12-16T11:13:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:13:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:13:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:13:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 6 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:13:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:13:08 2023 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:13:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap2-clean ']' 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap2-clean 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:57.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:57.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap3 2023-12-16T11:13:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap3 2023-12-16T11:13:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:13:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:13:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:13:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:13:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:13:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap3-clean 2023-12-16T11:13:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-16T11:13:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap3-clean 2023-12-16T11:13:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:13:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:13:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:13:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:13:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:13:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:13:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:13:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=6 2023-12-16T11:13:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:13:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:13:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 94296064 bytes 2023-12-16T11:13:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 127819776 bytes 2023-12-16T11:14:00.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 194334720 bytes 2023-12-16T11:14:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 246419456 bytes 2023-12-16T11:14:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 299459584 bytes 2023-12-16T11:14:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 360061952 bytes 2023-12-16T11:14:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 360657920 bytes 2023-12-16T11:14:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 366383104 bytes 2023-12-16T11:14:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 404006912 bytes 2023-12-16T11:14:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 405299200 bytes 2023-12-16T11:14:01.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 447212544 bytes 2023-12-16T11:14:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 481108992 bytes 2023-12-16T11:14:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 543858688 bytes 2023-12-16T11:14:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 567681024 bytes 2023-12-16T11:14:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 656166912 bytes 2023-12-16T11:14:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 679581696 bytes 2023-12-16T11:14:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 717150208 bytes 2023-12-16T11:14:02.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 764638208 bytes 2023-12-16T11:14:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 781675520 bytes 2023-12-16T11:14:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 789374976 bytes 2023-12-16T11:14:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 829640704 bytes 2023-12-16T11:14:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 831802368 bytes 2023-12-16T11:14:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 887360512 bytes 2023-12-16T11:14:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 998667264 bytes 2023-12-16T11:14:03.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1044866048 bytes 2023-12-16T11:14:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1083589632 bytes 2023-12-16T11:14:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1134701568 bytes 2023-12-16T11:14:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1150623744 bytes 2023-12-16T11:14:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1220165632 bytes 2023-12-16T11:14:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1239140352 bytes 2023-12-16T11:14:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1297878016 bytes 2023-12-16T11:14:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1386619904 bytes 2023-12-16T11:14:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1483898880 bytes 2023-12-16T11:14:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1518714880 bytes 2023-12-16T11:14:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1567761408 bytes 2023-12-16T11:14:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1581537280 bytes 2023-12-16T11:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1617221632 bytes 2023-12-16T11:14:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1673099264 bytes 2023-12-16T11:14:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1733392384 bytes 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:14:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:14:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:14:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:14:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:14:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:14:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:14:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:14:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:14:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535025664.0,"last_snapshot_bytes":535025664,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725182,"remote_snapshot_timestamp":1702725182,"replay_state":"idle"} 2023-12-16T11:14:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.2 on smithi160 2023-12-16T11:14:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:13:48 2023-12-16T11:14:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:14:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:14:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:14:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:14:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:13:48 2023-12-16T11:14:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:14:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535025664.0,"last_snapshot_bytes":535025664,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725182,"remote_snapshot_timestamp":1702725182,"replay_state":"idle"} 2023-12-16T11:14:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:14:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:14:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:14:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:14:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.2 on smithi160 2023-12-16T11:14:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:13:48' 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:13:48' 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:13:48' 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:14:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:14:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:14:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:14:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:14:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:14:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:14:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:13:48' 2023-12-16T11:14:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:14:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:14:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:14:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:14:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap4 2023-12-16T11:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap4 2023-12-16T11:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap4 2023-12-16T11:14:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:14:20.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9181077746410422) > weight (0.25) 2023-12-16T11:14:20.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4258948686760775) > weight (0.25) 2023-12-16T11:14:20.495 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9822393036774472) > weight (0.25) 2023-12-16T11:14:20.495 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.27711055329761325) > weight (0.25) 2023-12-16T11:14:20.495 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:14:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:14:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:14:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:14:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:14:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:14:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:14:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:14:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:14:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:14:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:14:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:14:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:14:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:14:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:14:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:14:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:14:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:14:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:14:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:14:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:14:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:14:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:14:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:14:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:14:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:14:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:14:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:14:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:14:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:14:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:14:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:14:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:14:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:14:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:14:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:14:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:14:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:14:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:14:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:14:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:14:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:14:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:14:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:14:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:14:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:14:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:14:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:14:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:14:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:14:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:14:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:14:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:14:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:14:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:14:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:14:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:14:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:14:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:14:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:14:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:14:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:14:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:14:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:14:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:14:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:14:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:14:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 17 2023-12-16T11:14:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:14:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:14:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:14:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:14:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:14:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=17 2023-12-16T11:14:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:14:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:14:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:14:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:14:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:14:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:14:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:14:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:14:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:14:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:14:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:14:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 17 = '' 2023-12-16T11:14:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:14:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:14:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:14:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:14:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:14:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:14:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:14:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:14:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:14:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:14:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 17 = '' 2023-12-16T11:14:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:14:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:14:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:14:40.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:14:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:14:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:14:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 17 = '' 2023-12-16T11:14:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:14:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:14:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:14:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:14:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:14:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:14:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=17 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 17 = 17 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap4 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:14:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:14:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-16T11:14:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:14:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 8 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:13:08 2023 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:14:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:14:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:14:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap3-clean ']' 2023-12-16T11:14:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap3-clean 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:14:43.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:14:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:14:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap4 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap4 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:14:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:14:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:14:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:14:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap4-clean 2023-12-16T11:14:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-16T11:14:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap4-clean 2023-12-16T11:14:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:14:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:14:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:14:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:14:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:14:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:14:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:14:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 32182272 bytes 2023-12-16T11:14:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 59834368 bytes 2023-12-16T11:14:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 93950976 bytes 2023-12-16T11:14:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 112083968 bytes 2023-12-16T11:14:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 117849088 bytes 2023-12-16T11:14:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 144796672 bytes 2023-12-16T11:14:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 184796160 bytes 2023-12-16T11:14:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 233900032 bytes 2023-12-16T11:14:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 300783616 bytes 2023-12-16T11:14:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 315222016 bytes 2023-12-16T11:14:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 384608256 bytes 2023-12-16T11:14:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 463255552 bytes 2023-12-16T11:14:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 471694336 bytes 2023-12-16T11:14:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 543950848 bytes 2023-12-16T11:14:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 554575872 bytes 2023-12-16T11:14:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 556894208 bytes 2023-12-16T11:14:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 630411264 bytes 2023-12-16T11:14:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 662474752 bytes 2023-12-16T11:14:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 700645376 bytes 2023-12-16T11:14:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 765001728 bytes 2023-12-16T11:14:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 818447360 bytes 2023-12-16T11:14:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 894398464 bytes 2023-12-16T11:14:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 950880256 bytes 2023-12-16T11:14:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 988940288 bytes 2023-12-16T11:14:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 999287808 bytes 2023-12-16T11:14:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1017806848 bytes 2023-12-16T11:14:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1023547392 bytes 2023-12-16T11:14:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1125516288 bytes 2023-12-16T11:14:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1194827776 bytes 2023-12-16T11:14:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1269928960 bytes 2023-12-16T11:14:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1289220096 bytes 2023-12-16T11:14:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1306041344 bytes 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:14:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:14:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:14:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:14:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:14:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:14:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:14:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:14:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:14:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":16673109.33,"bytes_per_snapshot":513256960.0,"last_snapshot_bytes":500193280,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725277,"remote_snapshot_timestamp":1702725277,"replay_state":"idle"} 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.2 on smithi160 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:14:48 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:14:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:14:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:14:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:14:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:14:48 2023-12-16T11:14:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:14:52.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":16673109.33,"bytes_per_snapshot":513256960.0,"last_snapshot_bytes":500193280,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725277,"remote_snapshot_timestamp":1702725277,"replay_state":"idle"} 2023-12-16T11:14:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:14:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:14:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:14:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:14:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.2 on smithi160 2023-12-16T11:14:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:14:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:14:48' 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:14:48' 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:14:48' 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:14:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:14:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:14:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:14:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:14:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:14:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:14:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:14:48' 2023-12-16T11:14:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:14:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:14:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:14:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:14:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:14:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap5 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap5 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap5 2023-12-16T11:14:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:15:08.501 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2842193025514197) > weight (0.25) 2023-12-16T11:15:08.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.354138524307286) > weight (0.25) 2023-12-16T11:15:08.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5216223911840293) > weight (0.25) 2023-12-16T11:15:08.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7066273116846612) > weight (0.25) 2023-12-16T11:15:08.502 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-16T11:15:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:15:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:15:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:15:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:15:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:15:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:15:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:15:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:15:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:15:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:15:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:15:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:15:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:15:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:15:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:15:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:15:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:15:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:15:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:15:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:15:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:15:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:15:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:15:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:15:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:15:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:15:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:15:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:15:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:15:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:15:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:15:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:15:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:15:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:15:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:15:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:15:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:15:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:15:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:15:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:15:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:15:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:15:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:15:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:15:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:15:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:15:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:15:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:15:22.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:15:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:15:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:15:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:15:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:15:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:15:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:15:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:15:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:15:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:15:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:15:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:15:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:15:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:15:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:15:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:15:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:15:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:15:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:15:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:15:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:15:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:15:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:15:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:15:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:15:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:15:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:15:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:15:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:15:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:15:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 21 2023-12-16T11:15:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:15:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:15:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:15:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:15:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:15:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:15:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=21 2023-12-16T11:15:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:15:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:15:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:15:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:15:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:15:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:15:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:15:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:15:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:15:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:15:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 21 = '' 2023-12-16T11:15:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:15:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:15:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:15:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:15:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:15:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:15:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:15:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:15:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:15:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:15:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 21 = '' 2023-12-16T11:15:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:15:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:15:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:15:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:15:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:15:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:15:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:15:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:15:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:15:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:15:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 21 = '' 2023-12-16T11:15:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:15:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:15:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:15:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:15:27.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:15:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=21 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 21 = 21 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap5 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:15:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:15:28.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-16T11:15:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:15:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:15:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:15:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 10 2023-12-16T11:15:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:15:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:15:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:15:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:15:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:15:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:15:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:15:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:14:43 2023 2023-12-16T11:15:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:15:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:15:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:15:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:15:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:15:28.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:15:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap4-clean ']' 2023-12-16T11:15:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-16T11:15:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap4-clean 2023-12-16T11:15:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:15:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:15:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:15:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:15:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:15:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap5 2023-12-16T11:15:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap5 2023-12-16T11:15:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:15:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:15:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:15:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:15:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:15:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap5-clean 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap5-clean 2023-12-16T11:15:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:15:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:15:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:15:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:15:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:15:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=10 2023-12-16T11:15:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:15:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:15:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 4603904 bytes 2023-12-16T11:15:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 73169920 bytes 2023-12-16T11:15:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 90119168 bytes 2023-12-16T11:15:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 127975424 bytes 2023-12-16T11:15:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 224721920 bytes 2023-12-16T11:15:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 301423616 bytes 2023-12-16T11:15:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 391397376 bytes 2023-12-16T11:15:33.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 404514816 bytes 2023-12-16T11:15:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 522145792 bytes 2023-12-16T11:15:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 538559488 bytes 2023-12-16T11:15:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 565157888 bytes 2023-12-16T11:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 630162432 bytes 2023-12-16T11:15:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 639886336 bytes 2023-12-16T11:15:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 734996480 bytes 2023-12-16T11:15:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 791410688 bytes 2023-12-16T11:15:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 829135872 bytes 2023-12-16T11:15:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 858600448 bytes 2023-12-16T11:15:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 901240832 bytes 2023-12-16T11:15:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 932459520 bytes 2023-12-16T11:15:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 980506624 bytes 2023-12-16T11:15:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1014935552 bytes 2023-12-16T11:15:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1044355072 bytes 2023-12-16T11:15:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1103542272 bytes 2023-12-16T11:15:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1105320960 bytes 2023-12-16T11:15:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1169484800 bytes 2023-12-16T11:15:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1193675776 bytes 2023-12-16T11:15:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1236381696 bytes 2023-12-16T11:15:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1263177728 bytes 2023-12-16T11:15:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1331542016 bytes 2023-12-16T11:15:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1397950464 bytes 2023-12-16T11:15:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1447997440 bytes 2023-12-16T11:15:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1484777472 bytes 2023-12-16T11:15:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1495683072 bytes 2023-12-16T11:15:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1533502464 bytes 2023-12-16T11:15:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1565091840 bytes 2023-12-16T11:15:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1650075648 bytes 2023-12-16T11:15:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1707403264 bytes 2023-12-16T11:15:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1741798400 bytes 2023-12-16T11:15:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1807916032 bytes 2023-12-16T11:15:38.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1853605888 bytes 2023-12-16T11:15:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1877672960 bytes 2023-12-16T11:15:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1897951232 bytes 2023-12-16T11:15:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1940973568 bytes 2023-12-16T11:15:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1974586368 bytes 2023-12-16T11:15:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1987742720 bytes 2023-12-16T11:15:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2011570176 bytes 2023-12-16T11:15:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2033092608 bytes 2023-12-16T11:15:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2063060992 bytes 2023-12-16T11:15:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2092160000 bytes 2023-12-16T11:15:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2125425664 bytes 2023-12-16T11:15:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2147330048 bytes 2023-12-16T11:15:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2205639680 bytes 2023-12-16T11:15:40.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2564742472910596) > weight (0.25) 2023-12-16T11:15:40.496 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-16T11:15:40.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-16T11:15:40.497 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-16T11:15:40.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:15:40.496+0000 7fb140d18640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 93985) UID: 1000 2023-12-16T11:15:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2277182464 bytes 2023-12-16T11:15:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2277817344 bytes 2023-12-16T11:15:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2298675200 bytes 2023-12-16T11:15:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2339449856 bytes 2023-12-16T11:15:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2379449344 bytes 2023-12-16T11:15:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2393502720 bytes 2023-12-16T11:15:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2419354624 bytes 2023-12-16T11:15:41.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2446564352 bytes 2023-12-16T11:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2491614208 bytes 2023-12-16T11:15:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2543773696 bytes 2023-12-16T11:15:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2584044544 bytes 2023-12-16T11:15:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2629047296 bytes 2023-12-16T11:15:42.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2686670848 bytes 2023-12-16T11:15:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2746776576 bytes 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:15:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:15:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:15:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:15:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:15:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:15:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":513256960.0,"last_snapshot_bytes":500193280,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725277,"remote_snapshot_timestamp":1702725277,"replay_state":"idle"} 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.2 on smithi160 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:15:18 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:15:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:15:18 2023-12-16T11:15:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:15:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":513256960.0,"last_snapshot_bytes":500193280,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725277,"remote_snapshot_timestamp":1702725277,"replay_state":"idle"} 2023-12-16T11:15:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:15:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:15:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:15:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:15:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.2 on smithi160 2023-12-16T11:15:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:15:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:15:18' 2023-12-16T11:15:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:15:18' 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:15:18' 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:15:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:15:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:15:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:15:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:15:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:15:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:15:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:15:18' 2023-12-16T11:15:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:15:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:15:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:15:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:15:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap6 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap6 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap6 2023-12-16T11:15:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:15:48.497 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-16T11:15:48.497 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:15:48.498 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-16T11:15:48.498 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-16T11:15:48.498 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-16T11:15:48.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-16T11:15:48.501 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-16T11:16:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:16:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:16:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:16:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:16:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:16:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:16:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:16:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:16:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:16:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:16:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:16:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:16:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:16:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:16:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:16:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:16:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:16:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:16:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:16:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:16:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:16:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:16:13.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:16:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:16:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:16:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:16:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:16:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:16:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:16:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:16:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:16:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:16:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:16:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:16:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:16:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:16:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:16:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:16:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:16:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:16:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:16:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:16:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:16:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:16:13.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:16:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:16:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:16:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:16:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:16:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:16:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:16:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:16:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:16:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:16:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:16:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:16:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:16:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:16:15.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 25 2023-12-16T11:16:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:16:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:16:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:16:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:16:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:16:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:16:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=25 2023-12-16T11:16:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:16:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:16:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:16:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:16:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:16:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:16:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:16:15.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:16:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:16:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:16:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:16:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 25 = '' 2023-12-16T11:16:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:16:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:16:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:16:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:16:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:16:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 25 = '' 2023-12-16T11:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:16:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:16:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:16:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:16:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:16:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:16:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:16:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:16:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:16:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:16:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 25 = '' 2023-12-16T11:16:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:16:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:16:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:16:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:16:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:16:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=25 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 25 = 25 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap6 2023-12-16T11:16:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:16:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:16:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:16:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 12 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:16:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:16:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:14:43 2023 2023-12-16T11:16:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:16:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:16:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:16:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:16:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:16:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:16:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap5-clean ']' 2023-12-16T11:16:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-16T11:16:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap5-clean 2023-12-16T11:16:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:16:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:16:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:16:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:16:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:16:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap6 2023-12-16T11:16:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap6 2023-12-16T11:16:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:16:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:16:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:16:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:16:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:16:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:23.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:16:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap6-clean 2023-12-16T11:16:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-16T11:16:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:16:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap6-clean 2023-12-16T11:16:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:16:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:16:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:16:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:16:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:16:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=13 2023-12-16T11:16:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:16:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:16:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 96211968 bytes 2023-12-16T11:16:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 147575808 bytes 2023-12-16T11:16:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 171343872 bytes 2023-12-16T11:16:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 198830080 bytes 2023-12-16T11:16:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 228984832 bytes 2023-12-16T11:16:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 246994944 bytes 2023-12-16T11:16:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 310948864 bytes 2023-12-16T11:16:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 330993664 bytes 2023-12-16T11:16:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 365864960 bytes 2023-12-16T11:16:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 384637952 bytes 2023-12-16T11:16:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 419189760 bytes 2023-12-16T11:16:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 464434176 bytes 2023-12-16T11:16:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 481356800 bytes 2023-12-16T11:16:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 501125120 bytes 2023-12-16T11:16:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 583203840 bytes 2023-12-16T11:16:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 643685376 bytes 2023-12-16T11:16:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 698653696 bytes 2023-12-16T11:16:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 774904832 bytes 2023-12-16T11:16:26.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3558441245280326) > weight (0.25) 2023-12-16T11:16:26.503 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.31912646062362904) > weight (0.25) 2023-12-16T11:16:26.503 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-16T11:16:26.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-16T11:16:26.503 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-16T11:16:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 837337088 bytes 2023-12-16T11:16:26.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:16:26.502+0000 7f962bad1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 91548) UID: 1000 2023-12-16T11:16:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 857432064 bytes 2023-12-16T11:16:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 895422464 bytes 2023-12-16T11:16:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 965651456 bytes 2023-12-16T11:16:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1019078656 bytes 2023-12-16T11:16:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1027094528 bytes 2023-12-16T11:16:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1035181056 bytes 2023-12-16T11:16:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1060800512 bytes 2023-12-16T11:16:27.504 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-16T11:16:27.504 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:16:27.504 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-16T11:16:27.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-16T11:16:27.505 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-16T11:16:27.507 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-16T11:16:27.507 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-16T11:16:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1103798272 bytes 2023-12-16T11:16:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1132750848 bytes 2023-12-16T11:16:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1184611328 bytes 2023-12-16T11:16:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1185259520 bytes 2023-12-16T11:16:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1205035008 bytes 2023-12-16T11:16:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1224869888 bytes 2023-12-16T11:16:28.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1261809664 bytes 2023-12-16T11:16:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1296811008 bytes 2023-12-16T11:16:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1301821440 bytes 2023-12-16T11:16:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1337953280 bytes 2023-12-16T11:16:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1370639360 bytes 2023-12-16T11:16:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1427875840 bytes 2023-12-16T11:16:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1534572544 bytes 2023-12-16T11:16:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1631343616 bytes 2023-12-16T11:16:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1711876096 bytes 2023-12-16T11:16:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1762756608 bytes 2023-12-16T11:16:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1811480576 bytes 2023-12-16T11:16:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1844884480 bytes 2023-12-16T11:16:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1848608768 bytes 2023-12-16T11:16:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1917973504 bytes 2023-12-16T11:16:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1945478144 bytes 2023-12-16T11:16:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1993293824 bytes 2023-12-16T11:16:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2073306112 bytes 2023-12-16T11:16:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2093717504 bytes 2023-12-16T11:16:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2151881728 bytes 2023-12-16T11:16:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2152515584 bytes 2023-12-16T11:16:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2183519232 bytes 2023-12-16T11:16:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2184127488 bytes 2023-12-16T11:16:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2217329664 bytes 2023-12-16T11:16:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2228030464 bytes 2023-12-16T11:16:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2251560960 bytes 2023-12-16T11:16:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2312060928 bytes 2023-12-16T11:16:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2321201152 bytes 2023-12-16T11:16:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2382554112 bytes 2023-12-16T11:16:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2432010240 bytes 2023-12-16T11:16:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2458575872 bytes 2023-12-16T11:16:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2506354688 bytes 2023-12-16T11:16:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2519873536 bytes 2023-12-16T11:16:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2544650240 bytes 2023-12-16T11:16:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2577958912 bytes 2023-12-16T11:16:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2641500160 bytes 2023-12-16T11:16:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2642349056 bytes 2023-12-16T11:16:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2699498496 bytes 2023-12-16T11:16:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2741246976 bytes 2023-12-16T11:16:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 2754853888 bytes 2023-12-16T11:16:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 2845782016 bytes 2023-12-16T11:16:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 2851385344 bytes 2023-12-16T11:16:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 2876342272 bytes 2023-12-16T11:16:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 2899208192 bytes 2023-12-16T11:16:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 2943757312 bytes 2023-12-16T11:16:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3005967360 bytes 2023-12-16T11:16:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3019963392 bytes 2023-12-16T11:16:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3044353024 bytes 2023-12-16T11:16:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8000 IOs, 3054715904 bytes 2023-12-16T11:16:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8100 IOs, 3078526976 bytes 2023-12-16T11:16:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8200 IOs, 3103153152 bytes 2023-12-16T11:16:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8300 IOs, 3163048960 bytes 2023-12-16T11:16:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8400 IOs, 3260380160 bytes 2023-12-16T11:16:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8500 IOs, 3284530176 bytes 2023-12-16T11:16:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8600 IOs, 3297880064 bytes 2023-12-16T11:16:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8700 IOs, 3302462464 bytes 2023-12-16T11:16:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8800 IOs, 3327805440 bytes 2023-12-16T11:16:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8900 IOs, 3357882368 bytes 2023-12-16T11:16:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:16:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:16:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:16:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:16:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:16:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:16:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:16:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:16:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:16:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:16:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:16:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:16:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:16:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:16:18 2023-12-16T11:16:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:16:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:16:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:16:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:16:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:16:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:16:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:16:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:16:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:16:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:16:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:16:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:16:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:16:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:16:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:16:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:16:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:16:18 2023-12-16T11:16:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:16:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:16:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:16:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:16:43.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:16:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:16:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:16:18 2023-12-16T11:16:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:16:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:16:43.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:16:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:16:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:16:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: down+unknown 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: status not found 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:16:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:16:48 2023-12-16T11:16:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 1 2023-12-16T11:16:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:16:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:16:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:16:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:16:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702725374,"remote_snapshot_timestamp":1702725374,"replay_state":"idle"} 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.3 on smithi160 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:16:57 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:16:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:16:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:16:48 2023-12-16T11:16:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:16:59.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702725374,"remote_snapshot_timestamp":1702725374,"replay_state":"idle"} 2023-12-16T11:16:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:16:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:16:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:16:59.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:16:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.3 on smithi160 2023-12-16T11:16:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:16:57' 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:16:57' 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:16:57' 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:16:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:16:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:16:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:16:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:16:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:16:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:16:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:16:57' 2023-12-16T11:16:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:16:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:16:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:16:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:16:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:16:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap7 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap7 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap7 2023-12-16T11:16:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:17:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:17:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:17:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:17:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:17:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:17:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:17:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:17:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:17:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:17:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:17:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:17:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:17:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:17:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:17:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:17:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:17:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:17:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:17:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:17:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:17:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:17:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:17:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:17:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:17:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:17:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:17:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:17:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:17:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:17:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:17:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:17:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:17:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:17:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:17:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:17:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:17:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:17:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:17:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:17:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:17:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:17:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:17:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:17:06.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:17:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:17:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:17:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:17:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:17:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:17:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:17:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:17:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:17:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:17:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:17:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:17:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:17:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:17:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:17:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:17:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:17:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:17:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:17:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:17:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:17:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:17:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:17:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 29 2023-12-16T11:17:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:17:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:17:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:17:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:17:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:17:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=29 2023-12-16T11:17:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:17:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:17:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:17:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:17:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:17:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:17:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:17:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:17:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:17:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:17:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:17:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 29 = '' 2023-12-16T11:17:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:17:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:17:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:17:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:17:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:17:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:17:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:17:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:17:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:17:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:17:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 29 = '' 2023-12-16T11:17:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:17:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:17:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:17:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:17:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:17:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:17:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:17:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:17:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:17:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:17:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 29 = '' 2023-12-16T11:17:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:17:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:17:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:17:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:17:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=29 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 29 = 29 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap7 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:17:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:17:12.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-16T11:17:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:17:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:17:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:17:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 14 2023-12-16T11:17:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:17:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:16:20 2023 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:17:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:17:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap6-clean ']' 2023-12-16T11:17:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap6-clean 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:17:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:17:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:17:14.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap7 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap7 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:17:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:17:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:17:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:17:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap7-clean 2023-12-16T11:17:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-16T11:17:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap7-clean 2023-12-16T11:17:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:17:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:17:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:17:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:17:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=9 2023-12-16T11:17:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:17:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:17:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 25893888 bytes 2023-12-16T11:17:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 67442688 bytes 2023-12-16T11:17:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 68057088 bytes 2023-12-16T11:17:18.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 114565120 bytes 2023-12-16T11:17:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 133689344 bytes 2023-12-16T11:17:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 191216640 bytes 2023-12-16T11:17:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 204170240 bytes 2023-12-16T11:17:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 319575040 bytes 2023-12-16T11:17:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 419139584 bytes 2023-12-16T11:17:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 456531968 bytes 2023-12-16T11:17:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 507416576 bytes 2023-12-16T11:17:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 516100096 bytes 2023-12-16T11:17:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 625165312 bytes 2023-12-16T11:17:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 661615616 bytes 2023-12-16T11:17:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 735282176 bytes 2023-12-16T11:17:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 797362176 bytes 2023-12-16T11:17:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 868803584 bytes 2023-12-16T11:17:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 950449152 bytes 2023-12-16T11:17:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 998473728 bytes 2023-12-16T11:17:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1040997376 bytes 2023-12-16T11:17:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1070855168 bytes 2023-12-16T11:17:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1103684608 bytes 2023-12-16T11:17:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1136694272 bytes 2023-12-16T11:17:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1168057344 bytes 2023-12-16T11:17:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1184050176 bytes 2023-12-16T11:17:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1255386112 bytes 2023-12-16T11:17:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1257122816 bytes 2023-12-16T11:17:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1335559168 bytes 2023-12-16T11:17:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1349191680 bytes 2023-12-16T11:17:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1379223552 bytes 2023-12-16T11:17:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1449916416 bytes 2023-12-16T11:17:22.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1552692224 bytes 2023-12-16T11:17:22.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1582668800 bytes 2023-12-16T11:17:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1613314048 bytes 2023-12-16T11:17:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1659719680 bytes 2023-12-16T11:17:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1691596800 bytes 2023-12-16T11:17:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1720891392 bytes 2023-12-16T11:17:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1784495104 bytes 2023-12-16T11:17:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1844724736 bytes 2023-12-16T11:17:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1904906240 bytes 2023-12-16T11:17:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1908265984 bytes 2023-12-16T11:17:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1943206912 bytes 2023-12-16T11:17:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1944636416 bytes 2023-12-16T11:17:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2012045312 bytes 2023-12-16T11:17:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2051365888 bytes 2023-12-16T11:17:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2125294592 bytes 2023-12-16T11:17:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2192154624 bytes 2023-12-16T11:17:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2221901824 bytes 2023-12-16T11:17:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2244564992 bytes 2023-12-16T11:17:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2280037376 bytes 2023-12-16T11:17:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2357025792 bytes 2023-12-16T11:17:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2400775168 bytes 2023-12-16T11:17:26.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2483005440 bytes 2023-12-16T11:17:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2498324480 bytes 2023-12-16T11:17:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2556755968 bytes 2023-12-16T11:17:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:17:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:17:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:17:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:17:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:17:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:17:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:17:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:17:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:17:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:17:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:17:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:17:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:17:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:17:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:17:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":26812313.6,"bytes_per_snapshot":536246272.0,"last_snapshot_bytes":536246272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725427,"remote_snapshot_timestamp":1702725427,"replay_state":"idle"} 2023-12-16T11:17:27.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.3 on smithi160 2023-12-16T11:17:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:17:18 2023-12-16T11:17:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:17:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:17:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:17:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:17:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:17:18 2023-12-16T11:17:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:17:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":26812313.6,"bytes_per_snapshot":536246272.0,"last_snapshot_bytes":536246272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725427,"remote_snapshot_timestamp":1702725427,"replay_state":"idle"} 2023-12-16T11:17:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:17:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:17:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:17:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:17:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.3 on smithi160 2023-12-16T11:17:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:17:18' 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:17:18' 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:17:18' 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:17:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:17:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:17:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:17:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:17:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:17:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:17:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:17:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:17:18' 2023-12-16T11:17:27.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:17:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:17:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:17:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:17:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:17:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap8 2023-12-16T11:17:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap8 2023-12-16T11:17:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap8 2023-12-16T11:17:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:17:30.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5256899166089312) > weight (0.25) 2023-12-16T11:17:30.503 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-16T11:17:30.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-16T11:17:30.503 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-16T11:17:30.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:17:30.502+0000 7f456ef7d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 96085) UID: 1000 2023-12-16T11:17:33.504 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-16T11:17:33.504 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:17:33.505 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-16T11:17:33.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-16T11:17:33.505 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-16T11:17:33.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-16T11:17:33.508 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-12-16T11:17:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:17:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:17:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:17:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:17:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:17:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:17:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:17:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:17:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:17:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:17:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:17:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:17:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:17:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:17:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:17:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:17:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:17:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:17:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:17:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:17:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:17:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:17:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:17:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:17:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:17:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:17:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:17:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:17:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:17:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:17:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:17:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:17:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:17:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:17:57.517 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8606942480567519) > weight (0.25) 2023-12-16T11:17:57.517 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-16T11:17:57.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-16T11:17:57.518 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-16T11:17:57.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:17:57.516+0000 7f87f5a43640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 93856) UID: 1000 2023-12-16T11:17:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:17:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:17:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:17:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:17:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:17:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:17:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:17:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:17:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:17:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:17:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:17:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:17:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:17:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:17:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:17:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:17:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:17:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:17:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:17:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:17:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:17:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:17:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:17:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:17:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:17:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:17:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:17:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:17:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:17:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:17:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:17:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:17:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:17:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:17:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:17:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:17:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:17:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:17:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:17:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:17:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:17:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:17:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:17:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:17:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:17:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:17:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:17:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 33 2023-12-16T11:17:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:17:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:17:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:17:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:17:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:17:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=33 2023-12-16T11:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:17:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:17:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:17:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:17:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:17:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:17:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:17:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:17:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:17:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:17:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:17:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:17:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:17:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 33 = '' 2023-12-16T11:17:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:17:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:18:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:18:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:18:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:18:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:18:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:18:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:18:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:18:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 33 = '' 2023-12-16T11:18:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:18:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:18:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:18:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:18:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:18:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:18:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:18:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 33 = '' 2023-12-16T11:18:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:18:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:18:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:18:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=33 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 33 = 33 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap8 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:18:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:18:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:18:03.519 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-16T11:18:03.519 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:18:03.519 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-16T11:18:03.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:18:03.519 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:18:03.522 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:18:03.522 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-12-16T11:18:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-16T11:18:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:18:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:18:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:18:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 16 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:16:20 2023 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:18:03.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap7-clean ']' 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap7-clean 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap8 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap8 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap8-clean 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap8-clean 2023-12-16T11:18:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:18:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:18:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:18:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:18:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=9 2023-12-16T11:18:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:18:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:18:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 57191424 bytes 2023-12-16T11:18:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 173717504 bytes 2023-12-16T11:18:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 206759936 bytes 2023-12-16T11:18:08.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 249657344 bytes 2023-12-16T11:18:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 279227392 bytes 2023-12-16T11:18:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 360598528 bytes 2023-12-16T11:18:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 446610432 bytes 2023-12-16T11:18:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 471797760 bytes 2023-12-16T11:18:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 561935360 bytes 2023-12-16T11:18:09.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 595844096 bytes 2023-12-16T11:18:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 616627200 bytes 2023-12-16T11:18:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 617238528 bytes 2023-12-16T11:18:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 668239872 bytes 2023-12-16T11:18:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 691155968 bytes 2023-12-16T11:18:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 715944960 bytes 2023-12-16T11:18:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 762615808 bytes 2023-12-16T11:18:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 778627072 bytes 2023-12-16T11:18:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 843512832 bytes 2023-12-16T11:18:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 880844800 bytes 2023-12-16T11:18:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 881472512 bytes 2023-12-16T11:18:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 908760064 bytes 2023-12-16T11:18:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 965174272 bytes 2023-12-16T11:18:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1022786560 bytes 2023-12-16T11:18:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1081213952 bytes 2023-12-16T11:18:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1137695744 bytes 2023-12-16T11:18:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1211328512 bytes 2023-12-16T11:18:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1299525632 bytes 2023-12-16T11:18:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1333956608 bytes 2023-12-16T11:18:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1365460992 bytes 2023-12-16T11:18:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1453065216 bytes 2023-12-16T11:18:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1517109248 bytes 2023-12-16T11:18:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1549498368 bytes 2023-12-16T11:18:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1559163904 bytes 2023-12-16T11:18:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1622056960 bytes 2023-12-16T11:18:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1670925312 bytes 2023-12-16T11:18:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1708857344 bytes 2023-12-16T11:18:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1769479168 bytes 2023-12-16T11:18:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1804119040 bytes 2023-12-16T11:18:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1859586048 bytes 2023-12-16T11:18:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1930963968 bytes 2023-12-16T11:18:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 2051673088 bytes 2023-12-16T11:18:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 2096881664 bytes 2023-12-16T11:18:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 2108700672 bytes 2023-12-16T11:18:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2130766848 bytes 2023-12-16T11:18:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2211232768 bytes 2023-12-16T11:18:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2244625408 bytes 2023-12-16T11:18:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2297238528 bytes 2023-12-16T11:18:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2318526464 bytes 2023-12-16T11:18:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2335975424 bytes 2023-12-16T11:18:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2351718400 bytes 2023-12-16T11:18:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2384046080 bytes 2023-12-16T11:18:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2430267392 bytes 2023-12-16T11:18:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2458658816 bytes 2023-12-16T11:18:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2538088448 bytes 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:18:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:18:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:18:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:18:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:18:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:18:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:18:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:18:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:18:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:18:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:18:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:18:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536246272.0,"last_snapshot_bytes":536246272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725427,"remote_snapshot_timestamp":1702725427,"replay_state":"idle"} 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.3 on smithi160 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:17:48 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:18:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:17:48 2023-12-16T11:18:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:18:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536246272.0,"last_snapshot_bytes":536246272,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725427,"remote_snapshot_timestamp":1702725427,"replay_state":"idle"} 2023-12-16T11:18:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:18:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:18:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:18:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:18:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.3 on smithi160 2023-12-16T11:18:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:17:48' 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:17:48' 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:17:48' 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:18:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:18:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:18:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:18:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:18:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:18:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:18:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:17:48' 2023-12-16T11:18:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:18:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:18:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:18:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:18:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap9 2023-12-16T11:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap9 2023-12-16T11:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap9 2023-12-16T11:18:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:18:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:18:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:18:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:18:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:18:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:18:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:18:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:18:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:18:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:18:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:18:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:18:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:18:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:18:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:18:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:18:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:18:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:18:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:18:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:18:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:18:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:18:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:18:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:18:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:18:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:18:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:18:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:18:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:18:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:18:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:18:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:18:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:18:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:18:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:18:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:18:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:18:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:18:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:18:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:18:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:18:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:18:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:18:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:18:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:18:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:18:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:18:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:18:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:18:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:18:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:18:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:18:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:18:44.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:18:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:18:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:18:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:18:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:18:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:18:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:18:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:18:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:18:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:18:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:18:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:18:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:18:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:18:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:18:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:18:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:18:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:18:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:18:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 37 2023-12-16T11:18:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:18:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:18:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:18:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:18:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=37 2023-12-16T11:18:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:18:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:18:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:18:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:18:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:18:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:18:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:18:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:18:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:18:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:18:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 37 = '' 2023-12-16T11:18:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:18:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:18:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:18:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:18:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:18:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:18:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:18:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:18:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 37 = '' 2023-12-16T11:18:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:18:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:18:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:18:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:18:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:18:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:18:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:18:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:18:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:18:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 37 = '' 2023-12-16T11:18:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:18:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:18:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:18:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:18:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:18:49.902 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:18:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=37 2023-12-16T11:18:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 37 = 37 2023-12-16T11:18:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-16T11:18:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:18:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap9 2023-12-16T11:18:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:18:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:18:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:18:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 18 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:18:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:18:04 2023 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:18:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap8-clean ']' 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap8-clean 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap9 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap9 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:18:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:18:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:18:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap9-clean 2023-12-16T11:18:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-16T11:18:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap9-clean 2023-12-16T11:18:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:18:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:18:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror test 2023-12-16T11:18:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:18:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:18:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:18:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:18:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:18:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:18:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-16T11:18:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 52884480 bytes 2023-12-16T11:18:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 67949568 bytes 2023-12-16T11:18:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 110866432 bytes 2023-12-16T11:18:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 138949632 bytes 2023-12-16T11:18:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 176813056 bytes 2023-12-16T11:18:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 223666176 bytes 2023-12-16T11:18:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 304081920 bytes 2023-12-16T11:18:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 349833216 bytes 2023-12-16T11:18:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 413742080 bytes 2023-12-16T11:18:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 483292160 bytes 2023-12-16T11:18:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 544098304 bytes 2023-12-16T11:18:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 586356736 bytes 2023-12-16T11:18:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 695899136 bytes 2023-12-16T11:18:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 722676736 bytes 2023-12-16T11:18:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 774222848 bytes 2023-12-16T11:18:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 803331072 bytes 2023-12-16T11:18:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 875805696 bytes 2023-12-16T11:18:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 946869248 bytes 2023-12-16T11:18:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 966714368 bytes 2023-12-16T11:18:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1012689920 bytes 2023-12-16T11:18:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1090646016 bytes 2023-12-16T11:18:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1126821888 bytes 2023-12-16T11:18:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1193183232 bytes 2023-12-16T11:18:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1219412992 bytes 2023-12-16T11:18:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1231917056 bytes 2023-12-16T11:18:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1232546816 bytes 2023-12-16T11:18:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1272039424 bytes 2023-12-16T11:18:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1317296128 bytes 2023-12-16T11:18:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1409505280 bytes 2023-12-16T11:18:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1453490176 bytes 2023-12-16T11:18:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1484220416 bytes 2023-12-16T11:18:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1533152256 bytes 2023-12-16T11:19:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1545122816 bytes 2023-12-16T11:19:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1581173760 bytes 2023-12-16T11:19:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1627789312 bytes 2023-12-16T11:19:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1731496960 bytes 2023-12-16T11:19:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1788545024 bytes 2023-12-16T11:19:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1809720320 bytes 2023-12-16T11:19:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1874415616 bytes 2023-12-16T11:19:01.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1886315520 bytes 2023-12-16T11:19:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1950277632 bytes 2023-12-16T11:19:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1961587712 bytes 2023-12-16T11:19:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1985902592 bytes 2023-12-16T11:19:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2035630080 bytes 2023-12-16T11:19:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2147580928 bytes 2023-12-16T11:19:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2234590208 bytes 2023-12-16T11:19:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2294380544 bytes 2023-12-16T11:19:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2309447680 bytes 2023-12-16T11:19:02.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2328325120 bytes 2023-12-16T11:19:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2416987136 bytes 2023-12-16T11:19:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2488792064 bytes 2023-12-16T11:19:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2522217472 bytes 2023-12-16T11:19:03.509 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7029569509750758) > weight (0.25) 2023-12-16T11:19:03.509 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9138689722246377) > weight (0.25) 2023-12-16T11:19:03.509 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-16T11:19:03.510 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-16T11:19:03.510 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-16T11:19:03.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:19:03.508+0000 7fc4eb1d6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 91699) UID: 1000 2023-12-16T11:19:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2573449216 bytes 2023-12-16T11:19:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2663148544 bytes 2023-12-16T11:19:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2744604672 bytes 2023-12-16T11:19:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2758296576 bytes 2023-12-16T11:19:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2794577920 bytes 2023-12-16T11:19:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2882600960 bytes 2023-12-16T11:19:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 3026358272 bytes 2023-12-16T11:19:05.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 3051164672 bytes 2023-12-16T11:19:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 3078165504 bytes 2023-12-16T11:19:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 3118274560 bytes 2023-12-16T11:19:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 3141246976 bytes 2023-12-16T11:19:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 3169537024 bytes 2023-12-16T11:19:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 3221974016 bytes 2023-12-16T11:19:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 3279024128 bytes 2023-12-16T11:19:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 3295926272 bytes 2023-12-16T11:19:06.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 3339627520 bytes 2023-12-16T11:19:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 3422692352 bytes 2023-12-16T11:19:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 3433682944 bytes 2023-12-16T11:19:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:19:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:19:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:19:07.511 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-16T11:19:07.511 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:19:07.511 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-16T11:19:07.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-16T11:19:07.511 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-16T11:19:07.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-16T11:19:07.514 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-12-16T11:19:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-12-16T11:19:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local description_pattern= 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local service_pattern= 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:19:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:19:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-12-16T11:19:07.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:test: 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: global_id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+replaying 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530092544.0,"last_snapshot_bytes":523938816,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725478,"remote_snapshot_timestamp":1702725525,"replay_state":"syncing","seconds_until_synced":162623581452388640,"syncing_percent":0,"syncing_snapshot_timestamp":1702725525} 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: service: mirror.3 on smithi160 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:18:48 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: peer_sites: 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: name: cluster2 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: state: up+stopped 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: description: local image is primary 2023-12-16T11:19:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: last_update: 2023-12-16 11:18:48 2023-12-16T11:19:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: state: up+replaying 2023-12-16T11:19:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^ description: .*' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":530092544.0,"last_snapshot_bytes":523938816,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702725478,"remote_snapshot_timestamp":1702725525,"replay_state":"syncing","seconds_until_synced":162623581452388640,"syncing_percent":0,"syncing_snapshot_timestamp":1702725525} 2023-12-16T11:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep '^up+' 2023-12-16T11:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: service: mirror.3 on smithi160 2023-12-16T11:19:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.mirror_status 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'last_update=2023-12-16 11:18:48' 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-12-16 11:18:48' 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state_pattern=up+replaying 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local 'prev_last_update=2023-12-16 11:18:48' 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror.mirror_status 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror.mirror_status 2023-12-16T11:19:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:19:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror.mirror_status 2023-12-16T11:19:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-16T11:19:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:19:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_update state 2023-12-16T11:19:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-12-16T11:19:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ last_update='2023-12-16 11:18:48' 2023-12-16T11:19:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-12-16T11:19:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=up+replaying 2023-12-16T11:19:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo up+replaying 2023-12-16T11:19:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep up+replaying 2023-12-16T11:19:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:up+replaying 2023-12-16T11:19:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap10 2023-12-16T11:19:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap10 2023-12-16T11:19:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap10 2023-12-16T11:19:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:19:25.523 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3107127419819973) > weight (0.25) 2023-12-16T11:19:25.524 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8675573636248958) > weight (0.25) 2023-12-16T11:19:25.524 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5093226713583666) > weight (0.25) 2023-12-16T11:19:25.524 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.4731719534659523) > weight (0.25) 2023-12-16T11:19:25.524 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-16T11:19:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:19:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:19:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:19:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:19:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:19:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:19:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:19:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:19:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:19:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:19:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:19:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:19:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:19:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:19:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:19:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:19:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:36.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:19:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:19:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:19:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:19:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:19:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:19:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:19:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:19:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:19:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:19:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:19:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:19:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:19:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:19:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:19:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:19:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:19:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:19:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:19:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:19:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:19:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:19:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:19:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:19:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:19:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:19:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:19:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:19:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:19:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:19:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:19:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:19:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:19:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:19:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:19:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:19:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:19:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:19:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:19:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:19:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:19:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:19:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:19:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:19:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:19:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:19:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:19:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:19:36.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:19:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:19:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:36.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:19:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 41 2023-12-16T11:19:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:19:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:19:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:19:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:19:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=41 2023-12-16T11:19:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:19:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:19:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:19:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:19:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:19:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:19:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:19:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:19:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 41 = '' 2023-12-16T11:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:19:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:19:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:19:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:19:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:19:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:19:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:19:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:19:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 41 = '' 2023-12-16T11:19:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:19:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:19:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:19:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:19:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:19:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:19:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:19:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:19:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:19:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 41 = '' 2023-12-16T11:19:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:19:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:19:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:19:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=41 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 41 = 41 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap10 2023-12-16T11:19:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:19:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:19:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:19:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 20 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:19:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:18:04 2023 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:19:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:19:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snap9-clean ']' 2023-12-16T11:19:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-16T11:19:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:42.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap9-clean 2023-12-16T11:19:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:19:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap10 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap10 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ clean_snap_name=snap10-clean 2023-12-16T11:19:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-16T11:19:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:46.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap10-clean 2023-12-16T11:19:46.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-16T11:19:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-16T11:19:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:19:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:19:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-16T11:19:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:19:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:19:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:19:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-16T11:19:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:19:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:19:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:19:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:19:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:19:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:19:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:19:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:19:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:19:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-16T11:19:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:19:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:19:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:19:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:19:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:19:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:19:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:19:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:19:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-16T11:19:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:19:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:19:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:19:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:19:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:19:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:19:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:19:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:19:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:19:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:19:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:19:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-16T11:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:19:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:19:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:19:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:19:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:19:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:19:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:19:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:19:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:19:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-16T11:19:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-16T11:19:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:19:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:19:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:19:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:19:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:19:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:19:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:19:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:19:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:19:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:19:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:19:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:19:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:19:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-16T11:19:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:19:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:19:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:19:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:19:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:19:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:19:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:19:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:19:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-test.status 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-test.status 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-test.status 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-test.status 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-16T11:19:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 44 2023-12-16T11:19:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:19:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:19:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.status 2023-12-16T11:19:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-16T11:19:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:19:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=44 2023-12-16T11:19:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:19:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:19:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:19:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:19:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:19:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:19:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:19:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:19:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:19:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:19:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 44 = '' 2023-12-16T11:19:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:19:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:19:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.status 2023-12-16T11:19:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-16T11:19:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:19:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:19:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=44 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 44 = 44 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap10-clean 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:19:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:19:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-16T11:19:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'test': 2023-12-16T11:19:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:19:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:19:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 21 2023-12-16T11:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10baa2bdc422 2023-12-16T11:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10baa2bdc422 2023-12-16T11:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:19:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:19:43 2023 2023-12-16T11:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:11:20 2023 2023-12-16T11:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:19:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f48b5c43-91c9-4e14-942c-cad1fdf56218 2023-12-16T11:19:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:19:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 10 2023-12-16T11:19:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:19:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap1 2023-12-16T11:19:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap1 2023-12-16T11:19:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap1 2023-12-16T11:19:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:19:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:53.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap1-clean 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap1-clean 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap2 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap2 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap2 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:19:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap2-clean 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap2-clean 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:19:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:19:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap3 2023-12-16T11:19:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap3 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap3 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:19:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:19:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:19:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap3-clean 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap3-clean 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:02.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap4 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap4 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap4 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:03.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap4-clean 2023-12-16T11:20:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-16T11:20:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap4-clean 2023-12-16T11:20:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap5 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap5 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap5 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:07.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap5-clean 2023-12-16T11:20:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-16T11:20:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap5-clean 2023-12-16T11:20:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap6 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap6 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap6 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:10.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap6-clean 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap6-clean 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap7 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap7 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap7 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap7-clean 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap7-clean 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap8 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap8 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap8 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:18.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap8-clean 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap8-clean 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap9 2023-12-16T11:20:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap9 2023-12-16T11:20:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap9 2023-12-16T11:20:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap9-clean 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap9-clean 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap10 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap10 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap10 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap10-clean 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap10-clean 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export 2023-12-16T11:20:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-16T11:20:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-test.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-test.export 2023-12-16T11:20:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 10 2023-12-16T11:20:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap1 2023-12-16T11:20:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-16T11:20:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap1 2023-12-16T11:20:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-16T11:20:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap1-clean 2023-12-16T11:20:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-16T11:20:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap1-clean 2023-12-16T11:20:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-16T11:20:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap2 2023-12-16T11:20:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-16T11:20:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap2 2023-12-16T11:20:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-16T11:20:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap2-clean 2023-12-16T11:20:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-16T11:20:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap2-clean 2023-12-16T11:20:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-16T11:20:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap3 2023-12-16T11:20:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-16T11:20:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap3 2023-12-16T11:20:30.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-16T11:20:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap3-clean 2023-12-16T11:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-16T11:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap3-clean 2023-12-16T11:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-16T11:20:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap4 2023-12-16T11:20:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-16T11:20:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap4 2023-12-16T11:20:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-16T11:20:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap4-clean 2023-12-16T11:20:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-16T11:20:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap4-clean 2023-12-16T11:20:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-16T11:20:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap5 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap5 2023-12-16T11:20:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-16T11:20:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap5-clean 2023-12-16T11:20:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-16T11:20:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap5-clean 2023-12-16T11:20:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-16T11:20:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap6 2023-12-16T11:20:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-16T11:20:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap6 2023-12-16T11:20:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-16T11:20:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap6-clean 2023-12-16T11:20:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-16T11:20:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap6-clean 2023-12-16T11:20:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-16T11:20:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap7 2023-12-16T11:20:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-16T11:20:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap7 2023-12-16T11:20:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-16T11:20:39.525 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-16T11:20:39.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-16T11:20:39.526 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-16T11:20:39.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:20:39.524+0000 7fc5f5750640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 94676) UID: 1000 2023-12-16T11:20:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap7-clean 2023-12-16T11:20:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-16T11:20:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap7-clean 2023-12-16T11:20:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-16T11:20:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap8 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap8 2023-12-16T11:20:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-16T11:20:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap8-clean 2023-12-16T11:20:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-16T11:20:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap8-clean 2023-12-16T11:20:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-16T11:20:42.527 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-16T11:20:42.527 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:20:42.527 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-16T11:20:42.527 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-16T11:20:42.528 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-16T11:20:42.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-16T11:20:42.531 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-16T11:20:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap9 2023-12-16T11:20:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-16T11:20:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap9 2023-12-16T11:20:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-16T11:20:43.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap9-clean 2023-12-16T11:20:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-16T11:20:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap9-clean 2023-12-16T11:20:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-16T11:20:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap10 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap10 2023-12-16T11:20:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-16T11:20:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap10-clean 2023-12-16T11:20:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-16T11:20:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap10-clean 2023-12-16T11:20:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-16T11:20:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:20:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror test 2023-12-16T11:20:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:20:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:20:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror test 2023-12-16T11:20:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-16T11:20:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-16T11:20:46.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:20:46.787+0000 7f54be95a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d43b6ab860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:20:46.790 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:20:46.787+0000 7f54c4966640 -1 rbd::mirror::ImageReplayer: 0x55d43aabc500 [3/f48b5c43-91c9-4e14-942c-cad1fdf56218] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:20:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:20:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror test 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:20:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=test 2023-12-16T11:20:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-16T11:20:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:20:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-16T11:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:20:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=test 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror test 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:20:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=test 2023-12-16T11:20:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-12-16T11:20:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:20:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-16T11:20:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:20:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:20:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:20:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ testlog 'TEST: create many images' 2023-12-16T11:20:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee -a /tmp/tmp.jvOAE3Umds/rbd-mirror.test.log 2023-12-16T11:20:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ date '+%F %T' 2023-12-16T11:20:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo 2023-12-16 11:20:47 TEST: create many images 2023-12-16T11:20:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:2023-12-16 11:20:47 TEST: create many images 2023-12-16T11:20:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ snap_name=snap 2023-12-16T11:20:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 50 2023-12-16T11:20:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:20:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_1 2023-12-16T11:20:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-16T11:20:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:20:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:20:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:20:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-16T11:20:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:20:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:20:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:20:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:20:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:20:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2023-12-16T11:20:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:20:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-16T11:20:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:20:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-16T11:20:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_1': 2023-12-16T11:20:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:20:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:20:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:20:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12823d2b3ed 2023-12-16T11:20:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12823d2b3ed 2023-12-16T11:20:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:20:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:20:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:20:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:20:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:20:47 2023 2023-12-16T11:20:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:20:47 2023 2023-12-16T11:20:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:20:47 2023 2023-12-16T11:20:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:20:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:20:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3fc80ffb-aa2e-47fa-aeab-e20034fa097b 2023-12-16T11:20:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:20:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:20:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-16T11:20:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:20:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:20:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:20:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:20:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=13 2023-12-16T11:20:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:20:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:20:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 73314304 bytes 2023-12-16T11:20:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 160258048 bytes 2023-12-16T11:20:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 177086464 bytes 2023-12-16T11:20:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 177713152 bytes 2023-12-16T11:20:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 184295424 bytes 2023-12-16T11:20:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 217433088 bytes 2023-12-16T11:20:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 252753920 bytes 2023-12-16T11:20:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 275295232 bytes 2023-12-16T11:20:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 378184704 bytes 2023-12-16T11:20:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 395110400 bytes 2023-12-16T11:20:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 431977472 bytes 2023-12-16T11:20:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 519162880 bytes 2023-12-16T11:20:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 592366592 bytes 2023-12-16T11:20:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 641141760 bytes 2023-12-16T11:20:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 653475840 bytes 2023-12-16T11:20:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 734781440 bytes 2023-12-16T11:20:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 772453376 bytes 2023-12-16T11:20:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 849628160 bytes 2023-12-16T11:20:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 900031488 bytes 2023-12-16T11:20:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 907092992 bytes 2023-12-16T11:20:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 947841024 bytes 2023-12-16T11:20:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 987503616 bytes 2023-12-16T11:20:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1070822400 bytes 2023-12-16T11:20:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1095191552 bytes 2023-12-16T11:20:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1133558784 bytes 2023-12-16T11:20:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1165272064 bytes 2023-12-16T11:20:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1250561024 bytes 2023-12-16T11:20:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1268205568 bytes 2023-12-16T11:20:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1319018496 bytes 2023-12-16T11:20:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1397066752 bytes 2023-12-16T11:20:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1416636416 bytes 2023-12-16T11:20:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1442370560 bytes 2023-12-16T11:20:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1460077568 bytes 2023-12-16T11:20:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1460711424 bytes 2023-12-16T11:20:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1556586496 bytes 2023-12-16T11:20:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1583512576 bytes 2023-12-16T11:20:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1629340672 bytes 2023-12-16T11:20:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1674021888 bytes 2023-12-16T11:20:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1709640704 bytes 2023-12-16T11:20:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1728349184 bytes 2023-12-16T11:20:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1753185280 bytes 2023-12-16T11:20:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1768220672 bytes 2023-12-16T11:20:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1839219712 bytes 2023-12-16T11:20:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1887972352 bytes 2023-12-16T11:20:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1900385280 bytes 2023-12-16T11:20:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1923659776 bytes 2023-12-16T11:20:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2000945152 bytes 2023-12-16T11:20:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2025008128 bytes 2023-12-16T11:20:56.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2075988992 bytes 2023-12-16T11:20:56.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2119252992 bytes 2023-12-16T11:20:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2139797504 bytes 2023-12-16T11:20:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2175447040 bytes 2023-12-16T11:20:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2199947264 bytes 2023-12-16T11:20:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2225925120 bytes 2023-12-16T11:20:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2304299008 bytes 2023-12-16T11:20:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2330263552 bytes 2023-12-16T11:20:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2429193216 bytes 2023-12-16T11:20:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2458224640 bytes 2023-12-16T11:20:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2486435840 bytes 2023-12-16T11:20:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2544508928 bytes 2023-12-16T11:20:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2613666816 bytes 2023-12-16T11:20:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2647512064 bytes 2023-12-16T11:20:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2693761024 bytes 2023-12-16T11:20:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2708785152 bytes 2023-12-16T11:20:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2777249792 bytes 2023-12-16T11:20:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2803610624 bytes 2023-12-16T11:20:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2807701504 bytes 2023-12-16T11:20:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2828720128 bytes 2023-12-16T11:20:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2864355328 bytes 2023-12-16T11:21:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2984423424 bytes 2023-12-16T11:21:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 3052366848 bytes 2023-12-16T11:21:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 3097387008 bytes 2023-12-16T11:21:00.515 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-16T11:21:00.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-16T11:21:00.516 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-16T11:21:00.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:21:00.514+0000 7f54c5968640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 91630) UID: 1000 2023-12-16T11:21:00.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 3120344064 bytes 2023-12-16T11:21:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 3142886400 bytes 2023-12-16T11:21:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 3178056704 bytes 2023-12-16T11:21:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 3207278592 bytes 2023-12-16T11:21:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3263147008 bytes 2023-12-16T11:21:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3348270080 bytes 2023-12-16T11:21:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3408991232 bytes 2023-12-16T11:21:01.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8000 IOs, 3476467712 bytes 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_2 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:21:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:21:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:21:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:01.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2023-12-16T11:21:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-16T11:21:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:21:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_2': 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 128cc37fe9d2 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.128cc37fe9d2 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:21:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:01 2023 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:01 2023 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:01 2023 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 5c88cdc9-a15c-46ab-8224-c40a8d9af053 2023-12-16T11:21:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:21:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:21:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-16T11:21:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:21:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:21:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:21:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:21:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:21:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 100224000 bytes 2023-12-16T11:21:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 138384384 bytes 2023-12-16T11:21:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 157275136 bytes 2023-12-16T11:21:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 199030784 bytes 2023-12-16T11:21:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 208317440 bytes 2023-12-16T11:21:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 222792704 bytes 2023-12-16T11:21:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 262747136 bytes 2023-12-16T11:21:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 369466368 bytes 2023-12-16T11:21:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 401234944 bytes 2023-12-16T11:21:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 431447040 bytes 2023-12-16T11:21:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 482018304 bytes 2023-12-16T11:21:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 541751296 bytes 2023-12-16T11:21:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 603910144 bytes 2023-12-16T11:21:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 627224576 bytes 2023-12-16T11:21:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 636350464 bytes 2023-12-16T11:21:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 641331200 bytes 2023-12-16T11:21:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 700737536 bytes 2023-12-16T11:21:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 741087232 bytes 2023-12-16T11:21:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 755743744 bytes 2023-12-16T11:21:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 776790016 bytes 2023-12-16T11:21:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 876093440 bytes 2023-12-16T11:21:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 896555008 bytes 2023-12-16T11:21:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 909225984 bytes 2023-12-16T11:21:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 918038528 bytes 2023-12-16T11:21:06.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 979942400 bytes 2023-12-16T11:21:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 986971136 bytes 2023-12-16T11:21:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1063690240 bytes 2023-12-16T11:21:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1097341952 bytes 2023-12-16T11:21:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1113442304 bytes 2023-12-16T11:21:07.516 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-16T11:21:07.517 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:21:07.517 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-16T11:21:07.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-16T11:21:07.517 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-16T11:21:07.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-16T11:21:07.520 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-16T11:21:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1167453184 bytes 2023-12-16T11:21:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1183453184 bytes 2023-12-16T11:21:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1247924224 bytes 2023-12-16T11:21:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1292774400 bytes 2023-12-16T11:21:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1332682752 bytes 2023-12-16T11:21:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1388488704 bytes 2023-12-16T11:21:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1460952064 bytes 2023-12-16T11:21:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1523765248 bytes 2023-12-16T11:21:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1558734848 bytes 2023-12-16T11:21:09.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1624382464 bytes 2023-12-16T11:21:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1719185408 bytes 2023-12-16T11:21:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1775646720 bytes 2023-12-16T11:21:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1867313152 bytes 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_3 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:21:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:21:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:21:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:21:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2023-12-16T11:21:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:21:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:21:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-16T11:21:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:21:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-16T11:21:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:21:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-16T11:21:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_3': 2023-12-16T11:21:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129fea284a9a 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129fea284a9a 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:10 2023 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:10 2023 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:10 2023 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:21:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:21:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3e35d21e-975d-43a4-a669-8eaeaf673ca7 2023-12-16T11:21:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:21:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:21:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-16T11:21:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:21:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:21:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:21:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:21:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:21:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 40842240 bytes 2023-12-16T11:21:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 70366208 bytes 2023-12-16T11:21:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 155402240 bytes 2023-12-16T11:21:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 255020032 bytes 2023-12-16T11:21:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 296623104 bytes 2023-12-16T11:21:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 308978688 bytes 2023-12-16T11:21:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 358132736 bytes 2023-12-16T11:21:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 404205568 bytes 2023-12-16T11:21:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 472081408 bytes 2023-12-16T11:21:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 505924608 bytes 2023-12-16T11:21:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 522220544 bytes 2023-12-16T11:21:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 557945856 bytes 2023-12-16T11:21:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 596415488 bytes 2023-12-16T11:21:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 651966464 bytes 2023-12-16T11:21:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 717862912 bytes 2023-12-16T11:21:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 730159104 bytes 2023-12-16T11:21:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 763169792 bytes 2023-12-16T11:21:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 796273664 bytes 2023-12-16T11:21:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 829642752 bytes 2023-12-16T11:21:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 843168768 bytes 2023-12-16T11:21:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 874038272 bytes 2023-12-16T11:21:14.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 933093376 bytes 2023-12-16T11:21:14.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 978868224 bytes 2023-12-16T11:21:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1001902080 bytes 2023-12-16T11:21:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1056050176 bytes 2023-12-16T11:21:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1081442304 bytes 2023-12-16T11:21:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1119247360 bytes 2023-12-16T11:21:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1141557248 bytes 2023-12-16T11:21:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1167808512 bytes 2023-12-16T11:21:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1204859904 bytes 2023-12-16T11:21:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1299778560 bytes 2023-12-16T11:21:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_4 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2023-12-16T11:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-16T11:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:21:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-16T11:21:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:21:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-16T11:21:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_4': 2023-12-16T11:21:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:21:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:21:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:21:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12a76dad3923 2023-12-16T11:21:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12a76dad3923 2023-12-16T11:21:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:15 2023 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:15 2023 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:15 2023 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f9aaccb8-5e33-401a-b44f-3e6c030d0568 2023-12-16T11:21:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:21:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:21:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-16T11:21:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:21:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:21:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=6 2023-12-16T11:21:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:21:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:21:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 16999424 bytes 2023-12-16T11:21:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 48428032 bytes 2023-12-16T11:21:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 124735488 bytes 2023-12-16T11:21:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 190852096 bytes 2023-12-16T11:21:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 227743744 bytes 2023-12-16T11:21:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 329449472 bytes 2023-12-16T11:21:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 378606592 bytes 2023-12-16T11:21:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 398613504 bytes 2023-12-16T11:21:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 439527424 bytes 2023-12-16T11:21:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 489264128 bytes 2023-12-16T11:21:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 576245760 bytes 2023-12-16T11:21:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 593925120 bytes 2023-12-16T11:21:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 626463744 bytes 2023-12-16T11:21:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 646688768 bytes 2023-12-16T11:21:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 658680832 bytes 2023-12-16T11:21:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 729911296 bytes 2023-12-16T11:21:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 765027328 bytes 2023-12-16T11:21:19.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 781538304 bytes 2023-12-16T11:21:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 822243328 bytes 2023-12-16T11:21:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 850762752 bytes 2023-12-16T11:21:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 895304704 bytes 2023-12-16T11:21:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 978750464 bytes 2023-12-16T11:21:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 986727424 bytes 2023-12-16T11:21:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 989993984 bytes 2023-12-16T11:21:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1037778944 bytes 2023-12-16T11:21:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1098635264 bytes 2023-12-16T11:21:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1119365120 bytes 2023-12-16T11:21:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1154558976 bytes 2023-12-16T11:21:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1228435456 bytes 2023-12-16T11:21:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1241793536 bytes 2023-12-16T11:21:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1312599040 bytes 2023-12-16T11:21:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1359824896 bytes 2023-12-16T11:21:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1385949184 bytes 2023-12-16T11:21:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1417288704 bytes 2023-12-16T11:21:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1436213248 bytes 2023-12-16T11:21:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1502257152 bytes 2023-12-16T11:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:21:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_5 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-16T11:21:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:21:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-16T11:21:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_5': 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b02c8a4ca2 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b02c8a4ca2 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:22 2023 2023-12-16T11:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:22 2023 2023-12-16T11:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:22 2023 2023-12-16T11:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 1402ce13-80d3-4f5c-82ee-6152013b9f92 2023-12-16T11:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:21:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:21:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-16T11:21:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:21:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:21:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=6 2023-12-16T11:21:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:21:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:21:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 89569280 bytes 2023-12-16T11:21:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 139412480 bytes 2023-12-16T11:21:23.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 140027904 bytes 2023-12-16T11:21:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 167718912 bytes 2023-12-16T11:21:23.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 207844352 bytes 2023-12-16T11:21:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 219383808 bytes 2023-12-16T11:21:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 292990976 bytes 2023-12-16T11:21:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 354828288 bytes 2023-12-16T11:21:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 485128192 bytes 2023-12-16T11:21:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 494146560 bytes 2023-12-16T11:21:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 510076928 bytes 2023-12-16T11:21:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 547551232 bytes 2023-12-16T11:21:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 630011904 bytes 2023-12-16T11:21:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 685293568 bytes 2023-12-16T11:21:25.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 740166656 bytes 2023-12-16T11:21:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 814327808 bytes 2023-12-16T11:21:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 855993344 bytes 2023-12-16T11:21:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 930693120 bytes 2023-12-16T11:21:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 958788608 bytes 2023-12-16T11:21:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 996836352 bytes 2023-12-16T11:21:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1010696192 bytes 2023-12-16T11:21:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1034476544 bytes 2023-12-16T11:21:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1080210432 bytes 2023-12-16T11:21:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1120376832 bytes 2023-12-16T11:21:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1154752512 bytes 2023-12-16T11:21:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1167380480 bytes 2023-12-16T11:21:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1178442752 bytes 2023-12-16T11:21:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1232466944 bytes 2023-12-16T11:21:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1255777280 bytes 2023-12-16T11:21:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1283142656 bytes 2023-12-16T11:21:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1363538944 bytes 2023-12-16T11:21:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1427819520 bytes 2023-12-16T11:21:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1450253312 bytes 2023-12-16T11:21:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1525446656 bytes 2023-12-16T11:21:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1627064320 bytes 2023-12-16T11:21:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1646284800 bytes 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_6 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:21:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:29.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:21:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-16T11:21:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_6': 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12ba7c3233ee 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12ba7c3233ee 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:29 2023 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:29 2023 2023-12-16T11:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:29 2023 2023-12-16T11:21:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:21:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:21:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: a5d4700f-a7ec-4855-83d5-735055fdfd2f 2023-12-16T11:21:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:21:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:21:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-16T11:21:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:21:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:21:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=14 2023-12-16T11:21:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:21:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:21:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 18118656 bytes 2023-12-16T11:21:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 69558272 bytes 2023-12-16T11:21:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 93812736 bytes 2023-12-16T11:21:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 183516160 bytes 2023-12-16T11:21:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 204109824 bytes 2023-12-16T11:21:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 241144832 bytes 2023-12-16T11:21:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 261390336 bytes 2023-12-16T11:21:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 361282560 bytes 2023-12-16T11:21:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 440583168 bytes 2023-12-16T11:21:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 516544512 bytes 2023-12-16T11:21:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 613818368 bytes 2023-12-16T11:21:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 631786496 bytes 2023-12-16T11:21:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 651891712 bytes 2023-12-16T11:21:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 679544832 bytes 2023-12-16T11:21:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 691510272 bytes 2023-12-16T11:21:32.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 724608000 bytes 2023-12-16T11:21:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 811144192 bytes 2023-12-16T11:21:32.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 826754048 bytes 2023-12-16T11:21:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 873475072 bytes 2023-12-16T11:21:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 998910976 bytes 2023-12-16T11:21:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1046070272 bytes 2023-12-16T11:21:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1046702080 bytes 2023-12-16T11:21:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1080206336 bytes 2023-12-16T11:21:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1127396352 bytes 2023-12-16T11:21:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1155996672 bytes 2023-12-16T11:21:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1245143040 bytes 2023-12-16T11:21:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1307909120 bytes 2023-12-16T11:21:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1347835904 bytes 2023-12-16T11:21:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1393958912 bytes 2023-12-16T11:21:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1394585600 bytes 2023-12-16T11:21:35.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1489447936 bytes 2023-12-16T11:21:35.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1527550976 bytes 2023-12-16T11:21:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1544443904 bytes 2023-12-16T11:21:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1602456576 bytes 2023-12-16T11:21:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1628393472 bytes 2023-12-16T11:21:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1715891200 bytes 2023-12-16T11:21:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1766422528 bytes 2023-12-16T11:21:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1803865088 bytes 2023-12-16T11:21:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1860976640 bytes 2023-12-16T11:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1909485568 bytes 2023-12-16T11:21:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1921059840 bytes 2023-12-16T11:21:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1976487936 bytes 2023-12-16T11:21:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 2017049600 bytes 2023-12-16T11:21:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2055345152 bytes 2023-12-16T11:21:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2143590400 bytes 2023-12-16T11:21:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2196130816 bytes 2023-12-16T11:21:37.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2256988160 bytes 2023-12-16T11:21:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2357862400 bytes 2023-12-16T11:21:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2358478848 bytes 2023-12-16T11:21:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2413661184 bytes 2023-12-16T11:21:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2456410112 bytes 2023-12-16T11:21:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2457003008 bytes 2023-12-16T11:21:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2485214208 bytes 2023-12-16T11:21:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2539929600 bytes 2023-12-16T11:21:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2582258688 bytes 2023-12-16T11:21:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2609738752 bytes 2023-12-16T11:21:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2668215296 bytes 2023-12-16T11:21:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2707825664 bytes 2023-12-16T11:21:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2774595584 bytes 2023-12-16T11:21:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2792731648 bytes 2023-12-16T11:21:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2880884736 bytes 2023-12-16T11:21:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2919326720 bytes 2023-12-16T11:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2938178560 bytes 2023-12-16T11:21:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2938796032 bytes 2023-12-16T11:21:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2954185728 bytes 2023-12-16T11:21:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2999563264 bytes 2023-12-16T11:21:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 3058758656 bytes 2023-12-16T11:21:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 3147255808 bytes 2023-12-16T11:21:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 3147863040 bytes 2023-12-16T11:21:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 3204205568 bytes 2023-12-16T11:21:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 3241320448 bytes 2023-12-16T11:21:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 3291610112 bytes 2023-12-16T11:21:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 3344966656 bytes 2023-12-16T11:21:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 3400158208 bytes 2023-12-16T11:21:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 3405677568 bytes 2023-12-16T11:21:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 3440734208 bytes 2023-12-16T11:21:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3510956032 bytes 2023-12-16T11:21:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3523311616 bytes 2023-12-16T11:21:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3590214656 bytes 2023-12-16T11:21:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8000 IOs, 3617448960 bytes 2023-12-16T11:21:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8100 IOs, 3682439168 bytes 2023-12-16T11:21:43.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8200 IOs, 3695344640 bytes 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_7 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-16T11:21:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:21:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:21:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:21:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:43.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-16T11:21:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:21:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_7': 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12c4696fc7ed 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12c4696fc7ed 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:21:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:43 2023 2023-12-16T11:21:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:43 2023 2023-12-16T11:21:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:43 2023 2023-12-16T11:21:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:21:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:21:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c054a432-d656-4791-bea1-c03a47bbbf82 2023-12-16T11:21:43.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:21:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:21:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-16T11:21:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:21:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:21:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=11 2023-12-16T11:21:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:21:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:21:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 62984192 bytes 2023-12-16T11:21:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 99780608 bytes 2023-12-16T11:21:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 107467776 bytes 2023-12-16T11:21:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 148169728 bytes 2023-12-16T11:21:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 178926592 bytes 2023-12-16T11:21:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 218102784 bytes 2023-12-16T11:21:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 237029376 bytes 2023-12-16T11:21:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 296077312 bytes 2023-12-16T11:21:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 344946688 bytes 2023-12-16T11:21:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 360566784 bytes 2023-12-16T11:21:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 429814784 bytes 2023-12-16T11:21:45.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 483396608 bytes 2023-12-16T11:21:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 506225664 bytes 2023-12-16T11:21:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 585559040 bytes 2023-12-16T11:21:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 595378176 bytes 2023-12-16T11:21:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 632152064 bytes 2023-12-16T11:21:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 674819072 bytes 2023-12-16T11:21:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 731918336 bytes 2023-12-16T11:21:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 768613376 bytes 2023-12-16T11:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 833610752 bytes 2023-12-16T11:21:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 852642816 bytes 2023-12-16T11:21:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 874494976 bytes 2023-12-16T11:21:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 947886080 bytes 2023-12-16T11:21:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 983710720 bytes 2023-12-16T11:21:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1051304960 bytes 2023-12-16T11:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1080728576 bytes 2023-12-16T11:21:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1125961728 bytes 2023-12-16T11:21:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1151389696 bytes 2023-12-16T11:21:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1162385408 bytes 2023-12-16T11:21:48.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1190617088 bytes 2023-12-16T11:21:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1217042432 bytes 2023-12-16T11:21:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1223454720 bytes 2023-12-16T11:21:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1302251520 bytes 2023-12-16T11:21:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1349097472 bytes 2023-12-16T11:21:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1391109120 bytes 2023-12-16T11:21:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1477517312 bytes 2023-12-16T11:21:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1538609152 bytes 2023-12-16T11:21:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1584867328 bytes 2023-12-16T11:21:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1621471232 bytes 2023-12-16T11:21:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1705199616 bytes 2023-12-16T11:21:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1731113984 bytes 2023-12-16T11:21:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1754664960 bytes 2023-12-16T11:21:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1817522176 bytes 2023-12-16T11:21:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1834620928 bytes 2023-12-16T11:21:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1842658304 bytes 2023-12-16T11:21:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1885752320 bytes 2023-12-16T11:21:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1942563840 bytes 2023-12-16T11:21:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1987265536 bytes 2023-12-16T11:21:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2024763392 bytes 2023-12-16T11:21:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2050588672 bytes 2023-12-16T11:21:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2058175488 bytes 2023-12-16T11:21:52.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2141923328 bytes 2023-12-16T11:21:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2168644608 bytes 2023-12-16T11:21:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2213596160 bytes 2023-12-16T11:21:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2238343168 bytes 2023-12-16T11:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2277609472 bytes 2023-12-16T11:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2309634048 bytes 2023-12-16T11:21:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2322387968 bytes 2023-12-16T11:21:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2353448960 bytes 2023-12-16T11:21:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2370353152 bytes 2023-12-16T11:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2390596608 bytes 2023-12-16T11:21:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2449364992 bytes 2023-12-16T11:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2466491392 bytes 2023-12-16T11:21:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2477168640 bytes 2023-12-16T11:21:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2527162368 bytes 2023-12-16T11:21:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2577452032 bytes 2023-12-16T11:21:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2629383168 bytes 2023-12-16T11:21:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:21:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_8 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:21:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:21:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-16T11:21:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:21:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:21:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:21:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:21:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:21:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2023-12-16T11:21:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:21:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:21:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:21:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-16T11:21:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:21:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:21:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-16T11:21:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:21:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-16T11:21:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_8': 2023-12-16T11:21:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:21:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:21:56.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:21:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12cc1ddb0efe 2023-12-16T11:21:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12cc1ddb0efe 2023-12-16T11:21:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:21:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:21:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:21:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:21:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:55 2023 2023-12-16T11:21:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:55 2023 2023-12-16T11:21:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:55 2023 2023-12-16T11:21:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:21:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:21:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 83b3cdc9-e375-4f49-874d-7fe1521a1bba 2023-12-16T11:21:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:21:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:21:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-16T11:21:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:21:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:21:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:21:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:21:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=8 2023-12-16T11:21:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:21:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:21:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 47235072 bytes 2023-12-16T11:21:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 66489344 bytes 2023-12-16T11:21:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 114251776 bytes 2023-12-16T11:21:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 143502336 bytes 2023-12-16T11:21:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 224329728 bytes 2023-12-16T11:21:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 275719168 bytes 2023-12-16T11:21:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 307638272 bytes 2023-12-16T11:21:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 310619136 bytes 2023-12-16T11:21:57.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 345185280 bytes 2023-12-16T11:21:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 365362176 bytes 2023-12-16T11:21:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 415693824 bytes 2023-12-16T11:21:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 470279168 bytes 2023-12-16T11:21:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 480696320 bytes 2023-12-16T11:21:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 535227392 bytes 2023-12-16T11:21:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 592351232 bytes 2023-12-16T11:21:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 688931840 bytes 2023-12-16T11:21:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 726866944 bytes 2023-12-16T11:21:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 745334784 bytes 2023-12-16T11:21:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 774491136 bytes 2023-12-16T11:21:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 826211328 bytes 2023-12-16T11:21:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 860050432 bytes 2023-12-16T11:21:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 923574272 bytes 2023-12-16T11:21:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 965810176 bytes 2023-12-16T11:21:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 983641088 bytes 2023-12-16T11:22:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1035163648 bytes 2023-12-16T11:22:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1067354112 bytes 2023-12-16T11:22:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1113133056 bytes 2023-12-16T11:22:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1172666368 bytes 2023-12-16T11:22:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1253437440 bytes 2023-12-16T11:22:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1314632704 bytes 2023-12-16T11:22:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1374631936 bytes 2023-12-16T11:22:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1457932288 bytes 2023-12-16T11:22:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1488377856 bytes 2023-12-16T11:22:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1557433344 bytes 2023-12-16T11:22:02.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1594242048 bytes 2023-12-16T11:22:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1667289088 bytes 2023-12-16T11:22:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1713888256 bytes 2023-12-16T11:22:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1790695424 bytes 2023-12-16T11:22:02.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1844000768 bytes 2023-12-16T11:22:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1898299392 bytes 2023-12-16T11:22:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1931425792 bytes 2023-12-16T11:22:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1972002816 bytes 2023-12-16T11:22:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1996650496 bytes 2023-12-16T11:22:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2051671040 bytes 2023-12-16T11:22:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2101228544 bytes 2023-12-16T11:22:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:22:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:22:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_9 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2023-12-16T11:22:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:22:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:22:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-16T11:22:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:22:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-16T11:22:04.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:22:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_9': 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12d52e542543 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12d52e542543 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:04 2023 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:04 2023 2023-12-16T11:22:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:04 2023 2023-12-16T11:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 73805cc2-ee84-44e2-99c5-7e848da6a3d9 2023-12-16T11:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:22:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:22:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-16T11:22:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:22:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:22:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=6 2023-12-16T11:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:22:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:22:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 69984256 bytes 2023-12-16T11:22:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 119267328 bytes 2023-12-16T11:22:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 177705984 bytes 2023-12-16T11:22:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 252323840 bytes 2023-12-16T11:22:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 320112640 bytes 2023-12-16T11:22:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 373981184 bytes 2023-12-16T11:22:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 437775360 bytes 2023-12-16T11:22:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 438383616 bytes 2023-12-16T11:22:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 496275456 bytes 2023-12-16T11:22:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 530515968 bytes 2023-12-16T11:22:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 599269376 bytes 2023-12-16T11:22:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 660899840 bytes 2023-12-16T11:22:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 713101312 bytes 2023-12-16T11:22:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 793553920 bytes 2023-12-16T11:22:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 836524032 bytes 2023-12-16T11:22:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 901125120 bytes 2023-12-16T11:22:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 941486080 bytes 2023-12-16T11:22:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 948093952 bytes 2023-12-16T11:22:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 970825728 bytes 2023-12-16T11:22:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1039604736 bytes 2023-12-16T11:22:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1117324288 bytes 2023-12-16T11:22:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1132085248 bytes 2023-12-16T11:22:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1148017664 bytes 2023-12-16T11:22:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1182120960 bytes 2023-12-16T11:22:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1212689408 bytes 2023-12-16T11:22:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1225237504 bytes 2023-12-16T11:22:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1273703424 bytes 2023-12-16T11:22:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1308039168 bytes 2023-12-16T11:22:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1374283776 bytes 2023-12-16T11:22:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1384517632 bytes 2023-12-16T11:22:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1408157696 bytes 2023-12-16T11:22:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1475343360 bytes 2023-12-16T11:22:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1493348352 bytes 2023-12-16T11:22:10.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1556344832 bytes 2023-12-16T11:22:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1576414208 bytes 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_10 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:22:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:22:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2023-12-16T11:22:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:22:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:22:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-16T11:22:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:22:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-16T11:22:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:22:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-16T11:22:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_10': 2023-12-16T11:22:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:22:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:22:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:22:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12de659ca448 2023-12-16T11:22:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12de659ca448 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:11 2023 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:11 2023 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:11 2023 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 23a9ee20-37a2-4d39-b844-a43578e6e5f2 2023-12-16T11:22:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:22:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:22:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-16T11:22:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:22:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:22:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=14 2023-12-16T11:22:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:22:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:22:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 28974080 bytes 2023-12-16T11:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 96873472 bytes 2023-12-16T11:22:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 159352832 bytes 2023-12-16T11:22:12.729 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:22:12.726+0000 7f13f7b15640 -1 reset not still connected to 0x5589c9db92b0 2023-12-16T11:22:12.731 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:22:12.727+0000 7f13f7b15640 -1 reset not still connected to 0x5589ca474d00 2023-12-16T11:22:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 246126592 bytes 2023-12-16T11:22:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 278162432 bytes 2023-12-16T11:22:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 300934144 bytes 2023-12-16T11:22:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 352403456 bytes 2023-12-16T11:22:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 371636224 bytes 2023-12-16T11:22:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 412606464 bytes 2023-12-16T11:22:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 447021056 bytes 2023-12-16T11:22:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 536693760 bytes 2023-12-16T11:22:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 580213760 bytes 2023-12-16T11:22:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 632388608 bytes 2023-12-16T11:22:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 653403136 bytes 2023-12-16T11:22:14.531 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3279765421311983) > weight (0.25) 2023-12-16T11:22:14.532 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6926357201081061) > weight (0.25) 2023-12-16T11:22:14.532 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7867758261020757) > weight (0.25) 2023-12-16T11:22:14.532 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8507936400845626) > weight (0.25) 2023-12-16T11:22:14.532 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2023-12-16T11:22:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 725260288 bytes 2023-12-16T11:22:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 810018816 bytes 2023-12-16T11:22:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 866713600 bytes 2023-12-16T11:22:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 876839936 bytes 2023-12-16T11:22:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 968263680 bytes 2023-12-16T11:22:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1007182848 bytes 2023-12-16T11:22:15.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1020078080 bytes 2023-12-16T11:22:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1065479168 bytes 2023-12-16T11:22:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1145903104 bytes 2023-12-16T11:22:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1181315072 bytes 2023-12-16T11:22:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1195524096 bytes 2023-12-16T11:22:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1196113920 bytes 2023-12-16T11:22:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1262896128 bytes 2023-12-16T11:22:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1280456704 bytes 2023-12-16T11:22:17.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1310396416 bytes 2023-12-16T11:22:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1311041536 bytes 2023-12-16T11:22:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1407785984 bytes 2023-12-16T11:22:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1469703168 bytes 2023-12-16T11:22:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1524912128 bytes 2023-12-16T11:22:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1576854528 bytes 2023-12-16T11:22:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1596897280 bytes 2023-12-16T11:22:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1610002432 bytes 2023-12-16T11:22:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1634369536 bytes 2023-12-16T11:22:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1688271872 bytes 2023-12-16T11:22:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1739521024 bytes 2023-12-16T11:22:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1781825536 bytes 2023-12-16T11:22:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1815690240 bytes 2023-12-16T11:22:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1850887168 bytes 2023-12-16T11:22:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1935919104 bytes 2023-12-16T11:22:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2003263488 bytes 2023-12-16T11:22:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2069744640 bytes 2023-12-16T11:22:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2099464192 bytes 2023-12-16T11:22:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2143498240 bytes 2023-12-16T11:22:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2174375936 bytes 2023-12-16T11:22:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2208154624 bytes 2023-12-16T11:22:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2313227264 bytes 2023-12-16T11:22:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2343878656 bytes 2023-12-16T11:22:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2362541056 bytes 2023-12-16T11:22:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2387288064 bytes 2023-12-16T11:22:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2408811520 bytes 2023-12-16T11:22:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2435173376 bytes 2023-12-16T11:22:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2489991168 bytes 2023-12-16T11:22:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2509056000 bytes 2023-12-16T11:22:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2535314432 bytes 2023-12-16T11:22:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2583366656 bytes 2023-12-16T11:22:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2623678464 bytes 2023-12-16T11:22:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2667750400 bytes 2023-12-16T11:22:22.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2704717824 bytes 2023-12-16T11:22:22.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2709524480 bytes 2023-12-16T11:22:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2778771456 bytes 2023-12-16T11:22:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2836499456 bytes 2023-12-16T11:22:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2881598464 bytes 2023-12-16T11:22:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2935845888 bytes 2023-12-16T11:22:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2973684736 bytes 2023-12-16T11:22:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 3021137920 bytes 2023-12-16T11:22:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 3111489536 bytes 2023-12-16T11:22:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 3112108032 bytes 2023-12-16T11:22:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 3127856128 bytes 2023-12-16T11:22:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 3193428992 bytes 2023-12-16T11:22:24.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 3288024064 bytes 2023-12-16T11:22:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 3305982976 bytes 2023-12-16T11:22:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 3401061376 bytes 2023-12-16T11:22:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3411124224 bytes 2023-12-16T11:22:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3478629376 bytes 2023-12-16T11:22:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3500194816 bytes 2023-12-16T11:22:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8000 IOs, 3519326208 bytes 2023-12-16T11:22:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:22:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:22:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:22:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_11 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-16T11:22:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2023-12-16T11:22:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:22:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-16T11:22:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:22:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-16T11:22:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_11': 2023-12-16T11:22:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12e626dd8f3f 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12e626dd8f3f 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:26 2023 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:26 2023 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:26 2023 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 05cc72d1-934d-49bc-8864-822da1eba303 2023-12-16T11:22:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:22:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:22:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-16T11:22:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:22:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:22:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=11 2023-12-16T11:22:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:22:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:22:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 38964224 bytes 2023-12-16T11:22:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 121398272 bytes 2023-12-16T11:22:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 149919744 bytes 2023-12-16T11:22:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 188050432 bytes 2023-12-16T11:22:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 214628352 bytes 2023-12-16T11:22:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 270772224 bytes 2023-12-16T11:22:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 285264896 bytes 2023-12-16T11:22:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 383954944 bytes 2023-12-16T11:22:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 432187392 bytes 2023-12-16T11:22:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 526564352 bytes 2023-12-16T11:22:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 564164608 bytes 2023-12-16T11:22:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 594136064 bytes 2023-12-16T11:22:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 621365248 bytes 2023-12-16T11:22:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 658034688 bytes 2023-12-16T11:22:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 738137088 bytes 2023-12-16T11:22:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 759605248 bytes 2023-12-16T11:22:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 760215552 bytes 2023-12-16T11:22:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 810534912 bytes 2023-12-16T11:22:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 828256256 bytes 2023-12-16T11:22:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 835992576 bytes 2023-12-16T11:22:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 907385856 bytes 2023-12-16T11:22:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 940312576 bytes 2023-12-16T11:22:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 961144832 bytes 2023-12-16T11:22:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1012160512 bytes 2023-12-16T11:22:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1028135936 bytes 2023-12-16T11:22:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1098439680 bytes 2023-12-16T11:22:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1126274048 bytes 2023-12-16T11:22:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1155524608 bytes 2023-12-16T11:22:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1210868736 bytes 2023-12-16T11:22:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1230711808 bytes 2023-12-16T11:22:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1267459072 bytes 2023-12-16T11:22:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1296581632 bytes 2023-12-16T11:22:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1329134592 bytes 2023-12-16T11:22:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1395122176 bytes 2023-12-16T11:22:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1423759360 bytes 2023-12-16T11:22:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1436474368 bytes 2023-12-16T11:22:33.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1470559232 bytes 2023-12-16T11:22:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1533976576 bytes 2023-12-16T11:22:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1534574592 bytes 2023-12-16T11:22:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1606671360 bytes 2023-12-16T11:22:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1664348160 bytes 2023-12-16T11:22:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1680028672 bytes 2023-12-16T11:22:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1701749760 bytes 2023-12-16T11:22:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1735389184 bytes 2023-12-16T11:22:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1797317632 bytes 2023-12-16T11:22:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1846119424 bytes 2023-12-16T11:22:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1887956992 bytes 2023-12-16T11:22:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1978348544 bytes 2023-12-16T11:22:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2036111360 bytes 2023-12-16T11:22:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2096485376 bytes 2023-12-16T11:22:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2127682560 bytes 2023-12-16T11:22:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2217053184 bytes 2023-12-16T11:22:35.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2271737856 bytes 2023-12-16T11:22:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2337170432 bytes 2023-12-16T11:22:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2417991680 bytes 2023-12-16T11:22:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2514418688 bytes 2023-12-16T11:22:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2539362304 bytes 2023-12-16T11:22:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2557247488 bytes 2023-12-16T11:22:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2586822656 bytes 2023-12-16T11:22:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2685315072 bytes 2023-12-16T11:22:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2722867200 bytes 2023-12-16T11:22:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2732423168 bytes 2023-12-16T11:22:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2799736832 bytes 2023-12-16T11:22:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2805180416 bytes 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_12 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:22:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:22:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:22:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:22:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2023-12-16T11:22:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-16T11:22:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:22:38.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-16T11:22:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_12': 2023-12-16T11:22:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12ee41cab601 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12ee41cab601 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:38 2023 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:38 2023 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:38 2023 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 81ebcd20-efb2-4bd1-b191-504c2f7127bc 2023-12-16T11:22:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-16T11:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:22:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:22:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=14 2023-12-16T11:22:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:22:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:22:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 28028928 bytes 2023-12-16T11:22:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 85762048 bytes 2023-12-16T11:22:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 150248448 bytes 2023-12-16T11:22:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 238058496 bytes 2023-12-16T11:22:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 274291712 bytes 2023-12-16T11:22:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 316368896 bytes 2023-12-16T11:22:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 358060032 bytes 2023-12-16T11:22:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 419605504 bytes 2023-12-16T11:22:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 420225024 bytes 2023-12-16T11:22:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 459070464 bytes 2023-12-16T11:22:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 515202048 bytes 2023-12-16T11:22:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 515787776 bytes 2023-12-16T11:22:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 548762624 bytes 2023-12-16T11:22:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 630408192 bytes 2023-12-16T11:22:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 728755200 bytes 2023-12-16T11:22:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 749877248 bytes 2023-12-16T11:22:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 786973696 bytes 2023-12-16T11:22:41.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 838629376 bytes 2023-12-16T11:22:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 888493056 bytes 2023-12-16T11:22:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 909578240 bytes 2023-12-16T11:22:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 969184256 bytes 2023-12-16T11:22:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1002617856 bytes 2023-12-16T11:22:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1051836416 bytes 2023-12-16T11:22:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1096041472 bytes 2023-12-16T11:22:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1108716544 bytes 2023-12-16T11:22:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1152824320 bytes 2023-12-16T11:22:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1161157632 bytes 2023-12-16T11:22:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1208828928 bytes 2023-12-16T11:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1266012160 bytes 2023-12-16T11:22:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1314487296 bytes 2023-12-16T11:22:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1319377920 bytes 2023-12-16T11:22:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1356900352 bytes 2023-12-16T11:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1377480704 bytes 2023-12-16T11:22:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1426032640 bytes 2023-12-16T11:22:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1452393472 bytes 2023-12-16T11:22:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1488293888 bytes 2023-12-16T11:22:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1540426752 bytes 2023-12-16T11:22:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1562525696 bytes 2023-12-16T11:22:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1610738688 bytes 2023-12-16T11:22:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1677277184 bytes 2023-12-16T11:22:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1683795968 bytes 2023-12-16T11:22:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1744563200 bytes 2023-12-16T11:22:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1773983744 bytes 2023-12-16T11:22:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1868848128 bytes 2023-12-16T11:22:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1869471744 bytes 2023-12-16T11:22:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1931354112 bytes 2023-12-16T11:22:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1949744128 bytes 2023-12-16T11:22:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1972795392 bytes 2023-12-16T11:22:47.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2010440704 bytes 2023-12-16T11:22:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2057264128 bytes 2023-12-16T11:22:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2105590784 bytes 2023-12-16T11:22:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2166179840 bytes 2023-12-16T11:22:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2191248384 bytes 2023-12-16T11:22:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2226166784 bytes 2023-12-16T11:22:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2280548352 bytes 2023-12-16T11:22:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2281183232 bytes 2023-12-16T11:22:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2286227456 bytes 2023-12-16T11:22:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2337367040 bytes 2023-12-16T11:22:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2387946496 bytes 2023-12-16T11:22:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2434967552 bytes 2023-12-16T11:22:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2489756672 bytes 2023-12-16T11:22:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2531225600 bytes 2023-12-16T11:22:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2554479616 bytes 2023-12-16T11:22:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2555107328 bytes 2023-12-16T11:22:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2590595072 bytes 2023-12-16T11:22:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2609881088 bytes 2023-12-16T11:22:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2693239808 bytes 2023-12-16T11:22:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2822261760 bytes 2023-12-16T11:22:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2913581056 bytes 2023-12-16T11:22:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2921144320 bytes 2023-12-16T11:22:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 2927687680 bytes 2023-12-16T11:22:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 2948454400 bytes 2023-12-16T11:22:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 2981146624 bytes 2023-12-16T11:22:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 3028532224 bytes 2023-12-16T11:22:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 3042540544 bytes 2023-12-16T11:22:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 3085080576 bytes 2023-12-16T11:22:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3124512768 bytes 2023-12-16T11:22:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3195018240 bytes 2023-12-16T11:22:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3248819200 bytes 2023-12-16T11:22:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8000 IOs, 3259378688 bytes 2023-12-16T11:22:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8100 IOs, 3337293824 bytes 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_13 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:22:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:22:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:22:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:22:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:22:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2023-12-16T11:22:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:22:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:22:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:22:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-16T11:22:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:22:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:22:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-16T11:22:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:22:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-16T11:22:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_13': 2023-12-16T11:22:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:22:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:22:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:22:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12f7745579d1 2023-12-16T11:22:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12f7745579d1 2023-12-16T11:22:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:52 2023 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:52 2023 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:52 2023 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 05362fb1-bf44-4319-b368-19d9f2aa9ac9 2023-12-16T11:22:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:22:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:22:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-16T11:22:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:22:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:22:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:22:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:22:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:22:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:22:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:22:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 12290048 bytes 2023-12-16T11:22:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 118142976 bytes 2023-12-16T11:22:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 248098816 bytes 2023-12-16T11:22:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 312954880 bytes 2023-12-16T11:22:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 325345280 bytes 2023-12-16T11:22:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 336633856 bytes 2023-12-16T11:22:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 337250304 bytes 2023-12-16T11:22:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 371668992 bytes 2023-12-16T11:22:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 410467328 bytes 2023-12-16T11:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 456634368 bytes 2023-12-16T11:22:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 510669824 bytes 2023-12-16T11:22:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 528575488 bytes 2023-12-16T11:22:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 612398080 bytes 2023-12-16T11:22:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 648387584 bytes 2023-12-16T11:22:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 710427648 bytes 2023-12-16T11:22:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 744771584 bytes 2023-12-16T11:22:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 789754880 bytes 2023-12-16T11:22:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 802716672 bytes 2023-12-16T11:22:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 841812992 bytes 2023-12-16T11:22:57.520 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-16T11:22:57.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-16T11:22:57.521 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-16T11:22:57.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:22:57.518+0000 7f6ad41dd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 100550) UID: 1000 2023-12-16T11:22:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 921380864 bytes 2023-12-16T11:22:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 979598336 bytes 2023-12-16T11:22:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1011670016 bytes 2023-12-16T11:22:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1035858944 bytes 2023-12-16T11:22:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1116988416 bytes 2023-12-16T11:22:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1163566080 bytes 2023-12-16T11:22:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1186445312 bytes 2023-12-16T11:22:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1189760000 bytes 2023-12-16T11:22:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1199840256 bytes 2023-12-16T11:22:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1225133056 bytes 2023-12-16T11:22:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1268142080 bytes 2023-12-16T11:22:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1309363200 bytes 2023-12-16T11:22:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1386169344 bytes 2023-12-16T11:22:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1443434496 bytes 2023-12-16T11:23:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1506469888 bytes 2023-12-16T11:23:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1558561792 bytes 2023-12-16T11:23:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1597828096 bytes 2023-12-16T11:23:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1621410816 bytes 2023-12-16T11:23:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1639726080 bytes 2023-12-16T11:23:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_14 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2023-12-16T11:23:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:23:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:23:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-16T11:23:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:23:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-16T11:23:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:23:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-16T11:23:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_14': 2023-12-16T11:23:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:23:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:23:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:23:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13007892687 2023-12-16T11:23:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13007892687 2023-12-16T11:23:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:00 2023 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:00 2023 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:00 2023 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 72b31ab9-6f6f-406d-aa42-5f6f80afc543 2023-12-16T11:23:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:23:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:23:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-16T11:23:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:23:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:23:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:23:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:23:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:23:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 46092288 bytes 2023-12-16T11:23:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 83345408 bytes 2023-12-16T11:23:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 115784704 bytes 2023-12-16T11:23:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 164470784 bytes 2023-12-16T11:23:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 194876416 bytes 2023-12-16T11:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 259424256 bytes 2023-12-16T11:23:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 324130816 bytes 2023-12-16T11:23:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 406184960 bytes 2023-12-16T11:23:02.522 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-16T11:23:02.522 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:23:02.523 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-16T11:23:02.523 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-16T11:23:02.523 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-16T11:23:02.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-16T11:23:02.527 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-16T11:23:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 417644544 bytes 2023-12-16T11:23:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 470491136 bytes 2023-12-16T11:23:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 516885504 bytes 2023-12-16T11:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 572610560 bytes 2023-12-16T11:23:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 640565248 bytes 2023-12-16T11:23:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 644132864 bytes 2023-12-16T11:23:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 704640000 bytes 2023-12-16T11:23:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 719580160 bytes 2023-12-16T11:23:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 768204800 bytes 2023-12-16T11:23:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 768821248 bytes 2023-12-16T11:23:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 770785280 bytes 2023-12-16T11:23:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 812340224 bytes 2023-12-16T11:23:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 848013312 bytes 2023-12-16T11:23:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 888809472 bytes 2023-12-16T11:23:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 924979200 bytes 2023-12-16T11:23:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 938655744 bytes 2023-12-16T11:23:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 984026112 bytes 2023-12-16T11:23:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1015341056 bytes 2023-12-16T11:23:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1103896576 bytes 2023-12-16T11:23:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1108228096 bytes 2023-12-16T11:23:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1153557504 bytes 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_15 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:23:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:23:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:23:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2023-12-16T11:23:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-16T11:23:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:23:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-16T11:23:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_15': 2023-12-16T11:23:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:23:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:23:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:23:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 131286b3b13d 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.131286b3b13d 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:06 2023 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:06 2023 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:06 2023 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 2bed7332-6b14-4681-8280-1b20686fbfe2 2023-12-16T11:23:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:23:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:23:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-16T11:23:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:23:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:23:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:23:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:23:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:23:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 63169536 bytes 2023-12-16T11:23:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 104052736 bytes 2023-12-16T11:23:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 134553600 bytes 2023-12-16T11:23:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 175566848 bytes 2023-12-16T11:23:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 198011904 bytes 2023-12-16T11:23:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 198768640 bytes 2023-12-16T11:23:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 272999424 bytes 2023-12-16T11:23:08.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 318869504 bytes 2023-12-16T11:23:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 398036992 bytes 2023-12-16T11:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 429607936 bytes 2023-12-16T11:23:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 494703616 bytes 2023-12-16T11:23:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 574538752 bytes 2023-12-16T11:23:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 607635456 bytes 2023-12-16T11:23:09.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 706115584 bytes 2023-12-16T11:23:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 791307264 bytes 2023-12-16T11:23:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 839069696 bytes 2023-12-16T11:23:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 856543232 bytes 2023-12-16T11:23:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 882232320 bytes 2023-12-16T11:23:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 949254144 bytes 2023-12-16T11:23:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 996088832 bytes 2023-12-16T11:23:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1028248576 bytes 2023-12-16T11:23:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1028847616 bytes 2023-12-16T11:23:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1042243584 bytes 2023-12-16T11:23:11.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1086202880 bytes 2023-12-16T11:23:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1111487488 bytes 2023-12-16T11:23:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1135538176 bytes 2023-12-16T11:23:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1218522112 bytes 2023-12-16T11:23:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1219163136 bytes 2023-12-16T11:23:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1223047168 bytes 2023-12-16T11:23:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1292116992 bytes 2023-12-16T11:23:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1332015104 bytes 2023-12-16T11:23:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1403703296 bytes 2023-12-16T11:23:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1433656320 bytes 2023-12-16T11:23:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1468609536 bytes 2023-12-16T11:23:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1542375424 bytes 2023-12-16T11:23:13.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1612624896 bytes 2023-12-16T11:23:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1664935936 bytes 2023-12-16T11:23:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1740600320 bytes 2023-12-16T11:23:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1769648128 bytes 2023-12-16T11:23:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1820780544 bytes 2023-12-16T11:23:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1821402112 bytes 2023-12-16T11:23:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1872219136 bytes 2023-12-16T11:23:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1921150976 bytes 2023-12-16T11:23:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1952620544 bytes 2023-12-16T11:23:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1966550016 bytes 2023-12-16T11:23:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1989924864 bytes 2023-12-16T11:23:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2023401472 bytes 2023-12-16T11:23:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2043693056 bytes 2023-12-16T11:23:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2050135040 bytes 2023-12-16T11:23:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2073698304 bytes 2023-12-16T11:23:15.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2076843008 bytes 2023-12-16T11:23:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2077446144 bytes 2023-12-16T11:23:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2127109120 bytes 2023-12-16T11:23:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2154910720 bytes 2023-12-16T11:23:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2238080000 bytes 2023-12-16T11:23:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2296385536 bytes 2023-12-16T11:23:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2315783168 bytes 2023-12-16T11:23:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2380809216 bytes 2023-12-16T11:23:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2405604352 bytes 2023-12-16T11:23:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2442523648 bytes 2023-12-16T11:23:17.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2502137856 bytes 2023-12-16T11:23:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2524739584 bytes 2023-12-16T11:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2590712832 bytes 2023-12-16T11:23:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2655373312 bytes 2023-12-16T11:23:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2705792000 bytes 2023-12-16T11:23:18.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2770061312 bytes 2023-12-16T11:23:18.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2770680832 bytes 2023-12-16T11:23:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2819916800 bytes 2023-12-16T11:23:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2862841856 bytes 2023-12-16T11:23:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2944099328 bytes 2023-12-16T11:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:23:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_16 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:23:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-16T11:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:23:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:23:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:23:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:23:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2023-12-16T11:23:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-16T11:23:19.533 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7814414691533295) > weight (0.25) 2023-12-16T11:23:19.533 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-16T11:23:19.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-16T11:23:19.534 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-16T11:23:19.534 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:23:19.531+0000 7fc90d0ca640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 97533) UID: 1000 2023-12-16T11:23:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:23:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_16': 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 131b8d4c6990 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.131b8d4c6990 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:23:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:19 2023 2023-12-16T11:23:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:19 2023 2023-12-16T11:23:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:19 2023 2023-12-16T11:23:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:23:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:23:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 6981c0f3-5872-4bbd-931e-0d0c2fde30ef 2023-12-16T11:23:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:23:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:23:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-16T11:23:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:23:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:23:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=9 2023-12-16T11:23:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:23:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:23:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 628736 bytes 2023-12-16T11:23:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 40458240 bytes 2023-12-16T11:23:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 78858240 bytes 2023-12-16T11:23:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 120485888 bytes 2023-12-16T11:23:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 141219840 bytes 2023-12-16T11:23:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 239118336 bytes 2023-12-16T11:23:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 279848960 bytes 2023-12-16T11:23:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 287407104 bytes 2023-12-16T11:23:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 318322688 bytes 2023-12-16T11:23:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 374644736 bytes 2023-12-16T11:23:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 395119616 bytes 2023-12-16T11:23:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 438434816 bytes 2023-12-16T11:23:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 455563264 bytes 2023-12-16T11:23:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 538067968 bytes 2023-12-16T11:23:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 582294528 bytes 2023-12-16T11:23:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 617496576 bytes 2023-12-16T11:23:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 687382528 bytes 2023-12-16T11:23:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 718751744 bytes 2023-12-16T11:23:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 794686464 bytes 2023-12-16T11:23:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 808388608 bytes 2023-12-16T11:23:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 821097472 bytes 2023-12-16T11:23:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 869354496 bytes 2023-12-16T11:23:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 897316864 bytes 2023-12-16T11:23:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 978204672 bytes 2023-12-16T11:23:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1003947008 bytes 2023-12-16T11:23:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1041478656 bytes 2023-12-16T11:23:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1070965760 bytes 2023-12-16T11:23:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1116915712 bytes 2023-12-16T11:23:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1169338368 bytes 2023-12-16T11:23:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1171478528 bytes 2023-12-16T11:23:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1213740032 bytes 2023-12-16T11:23:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1283096576 bytes 2023-12-16T11:23:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1291527168 bytes 2023-12-16T11:23:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1337679872 bytes 2023-12-16T11:23:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1365410816 bytes 2023-12-16T11:23:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1391238144 bytes 2023-12-16T11:23:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1487571968 bytes 2023-12-16T11:23:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1507008512 bytes 2023-12-16T11:23:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1533350912 bytes 2023-12-16T11:23:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1572654080 bytes 2023-12-16T11:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1618238464 bytes 2023-12-16T11:23:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1643674624 bytes 2023-12-16T11:23:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1680520192 bytes 2023-12-16T11:23:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1699564544 bytes 2023-12-16T11:23:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1729099776 bytes 2023-12-16T11:23:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1744309248 bytes 2023-12-16T11:23:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1793887232 bytes 2023-12-16T11:23:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1799971840 bytes 2023-12-16T11:23:27.535 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-16T11:23:27.535 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:23:27.535 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-16T11:23:27.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:23:27.535 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:23:27.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:23:27.538 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-12-16T11:23:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1821927424 bytes 2023-12-16T11:23:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1869366272 bytes 2023-12-16T11:23:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1904831488 bytes 2023-12-16T11:23:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 1955426304 bytes 2023-12-16T11:23:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 1964667904 bytes 2023-12-16T11:23:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2027999232 bytes 2023-12-16T11:23:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2052612096 bytes 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_17 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-16T11:23:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:23:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:23:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:23:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:28.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-16T11:23:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:23:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-16T11:23:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_17': 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 132cc3f76bd7 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.132cc3f76bd7 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:28 2023 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:28 2023 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:28 2023 2023-12-16T11:23:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:23:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:23:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 476836f7-4398-4e1c-91b2-edee8e248309 2023-12-16T11:23:29.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:23:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:23:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-16T11:23:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:23:29.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:23:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=9 2023-12-16T11:23:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:23:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:23:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 28466176 bytes 2023-12-16T11:23:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 92209152 bytes 2023-12-16T11:23:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 153115648 bytes 2023-12-16T11:23:30.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 227628032 bytes 2023-12-16T11:23:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 245976064 bytes 2023-12-16T11:23:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 332399616 bytes 2023-12-16T11:23:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 389492736 bytes 2023-12-16T11:23:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 409840640 bytes 2023-12-16T11:23:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 462300160 bytes 2023-12-16T11:23:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 462898176 bytes 2023-12-16T11:23:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 474167296 bytes 2023-12-16T11:23:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 507816960 bytes 2023-12-16T11:23:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 556345344 bytes 2023-12-16T11:23:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 616787968 bytes 2023-12-16T11:23:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 650991616 bytes 2023-12-16T11:23:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 701577216 bytes 2023-12-16T11:23:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 748112896 bytes 2023-12-16T11:23:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 760267776 bytes 2023-12-16T11:23:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 790553600 bytes 2023-12-16T11:23:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 804899840 bytes 2023-12-16T11:23:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 853430272 bytes 2023-12-16T11:23:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 873578496 bytes 2023-12-16T11:23:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 892957696 bytes 2023-12-16T11:23:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 910031872 bytes 2023-12-16T11:23:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 937915392 bytes 2023-12-16T11:23:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 948942848 bytes 2023-12-16T11:23:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 962207744 bytes 2023-12-16T11:23:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1011814400 bytes 2023-12-16T11:23:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1085785088 bytes 2023-12-16T11:23:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1093136384 bytes 2023-12-16T11:23:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1102605312 bytes 2023-12-16T11:23:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1120087040 bytes 2023-12-16T11:23:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1163067392 bytes 2023-12-16T11:23:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1186130944 bytes 2023-12-16T11:23:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1186726912 bytes 2023-12-16T11:23:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1214903296 bytes 2023-12-16T11:23:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1242530816 bytes 2023-12-16T11:23:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1317967872 bytes 2023-12-16T11:23:36.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1347217408 bytes 2023-12-16T11:23:36.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1403507712 bytes 2023-12-16T11:23:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1460995072 bytes 2023-12-16T11:23:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1489005568 bytes 2023-12-16T11:23:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1545698304 bytes 2023-12-16T11:23:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1546303488 bytes 2023-12-16T11:23:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1623853056 bytes 2023-12-16T11:23:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1680016384 bytes 2023-12-16T11:23:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1707779072 bytes 2023-12-16T11:23:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1749744640 bytes 2023-12-16T11:23:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1802976256 bytes 2023-12-16T11:23:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1841006592 bytes 2023-12-16T11:23:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1883890688 bytes 2023-12-16T11:23:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 1907955712 bytes 2023-12-16T11:23:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 1952837632 bytes 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_18 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:23:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:23:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2023-12-16T11:23:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:23:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:23:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-16T11:23:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:23:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-16T11:23:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:23:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-16T11:23:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_18': 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 133891cb1c4d 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.133891cb1c4d 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:38 2023 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:38 2023 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:38 2023 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:23:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c8047879-aa5e-417a-81ea-d0a79500d75f 2023-12-16T11:23:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:23:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:23:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-16T11:23:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:23:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:23:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:23:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:23:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:23:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 32829440 bytes 2023-12-16T11:23:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 76366848 bytes 2023-12-16T11:23:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 96636928 bytes 2023-12-16T11:23:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 126765056 bytes 2023-12-16T11:23:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 152963072 bytes 2023-12-16T11:23:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 262406144 bytes 2023-12-16T11:23:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 301395968 bytes 2023-12-16T11:23:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 311319552 bytes 2023-12-16T11:23:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 343062528 bytes 2023-12-16T11:23:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 399161344 bytes 2023-12-16T11:23:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 467916800 bytes 2023-12-16T11:23:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 556424192 bytes 2023-12-16T11:23:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 642842624 bytes 2023-12-16T11:23:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 715805696 bytes 2023-12-16T11:23:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 754450432 bytes 2023-12-16T11:23:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 799078400 bytes 2023-12-16T11:23:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 832504832 bytes 2023-12-16T11:23:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 853385216 bytes 2023-12-16T11:23:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 896001024 bytes 2023-12-16T11:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 970343424 bytes 2023-12-16T11:23:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1039783936 bytes 2023-12-16T11:23:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1073458176 bytes 2023-12-16T11:23:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1112715264 bytes 2023-12-16T11:23:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1153530880 bytes 2023-12-16T11:23:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1234584576 bytes 2023-12-16T11:23:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1259245568 bytes 2023-12-16T11:23:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1266451456 bytes 2023-12-16T11:23:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1281873920 bytes 2023-12-16T11:23:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1315913728 bytes 2023-12-16T11:23:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1355548672 bytes 2023-12-16T11:23:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1398620160 bytes 2023-12-16T11:23:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1408595968 bytes 2023-12-16T11:23:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1425466368 bytes 2023-12-16T11:23:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1458995200 bytes 2023-12-16T11:23:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1469663232 bytes 2023-12-16T11:23:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1505767424 bytes 2023-12-16T11:23:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1569646592 bytes 2023-12-16T11:23:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1585798144 bytes 2023-12-16T11:23:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1621360640 bytes 2023-12-16T11:23:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1699198976 bytes 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_19 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2023-12-16T11:23:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:23:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:23:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-16T11:23:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:23:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-16T11:23:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:23:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_19': 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13447ae94c20 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13447ae94c20 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:23:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:46 2023 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:46 2023 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:46 2023 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: b5372d96-8f2a-4940-b250-2dbc5ed7b604 2023-12-16T11:23:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:23:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:23:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-16T11:23:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:23:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:23:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=6 2023-12-16T11:23:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:23:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:23:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 52651008 bytes 2023-12-16T11:23:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 99454976 bytes 2023-12-16T11:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 123341824 bytes 2023-12-16T11:23:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 245670912 bytes 2023-12-16T11:23:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 257324032 bytes 2023-12-16T11:23:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 303437824 bytes 2023-12-16T11:23:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 366836736 bytes 2023-12-16T11:23:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 368695296 bytes 2023-12-16T11:23:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 379702272 bytes 2023-12-16T11:23:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 398850048 bytes 2023-12-16T11:23:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 507076608 bytes 2023-12-16T11:23:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 531128320 bytes 2023-12-16T11:23:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 570178560 bytes 2023-12-16T11:23:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 609487872 bytes 2023-12-16T11:23:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 727265280 bytes 2023-12-16T11:23:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 771021824 bytes 2023-12-16T11:23:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 832100352 bytes 2023-12-16T11:23:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 877449216 bytes 2023-12-16T11:23:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 919966720 bytes 2023-12-16T11:23:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 961924096 bytes 2023-12-16T11:23:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1008695296 bytes 2023-12-16T11:23:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1039876096 bytes 2023-12-16T11:23:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1104662528 bytes 2023-12-16T11:23:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1159180288 bytes 2023-12-16T11:23:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1186252800 bytes 2023-12-16T11:23:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1190816768 bytes 2023-12-16T11:23:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1232113664 bytes 2023-12-16T11:23:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1232759808 bytes 2023-12-16T11:23:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1302758400 bytes 2023-12-16T11:23:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1342647296 bytes 2023-12-16T11:23:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1368365056 bytes 2023-12-16T11:23:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1390555136 bytes 2023-12-16T11:23:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1447921664 bytes 2023-12-16T11:23:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1469744128 bytes 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_20 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:23:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:23:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:23:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:23:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:23:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:23:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-16T11:23:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:23:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-16T11:23:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_20': 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13504218d03b 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13504218d03b 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:53 2023 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:53 2023 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:53 2023 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:23:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:23:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: a2f9df53-7e43-494c-88e1-9a03c24ccda8 2023-12-16T11:23:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:23:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:23:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=13 2023-12-16T11:23:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:23:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:23:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 25221120 bytes 2023-12-16T11:23:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 72124416 bytes 2023-12-16T11:23:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 85206016 bytes 2023-12-16T11:23:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 150109184 bytes 2023-12-16T11:23:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 169529344 bytes 2023-12-16T11:23:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 237900800 bytes 2023-12-16T11:23:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 312246272 bytes 2023-12-16T11:23:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 343724032 bytes 2023-12-16T11:23:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 361897984 bytes 2023-12-16T11:23:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 383993856 bytes 2023-12-16T11:23:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 429660160 bytes 2023-12-16T11:23:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 493603840 bytes 2023-12-16T11:23:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 535206912 bytes 2023-12-16T11:23:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 592311296 bytes 2023-12-16T11:23:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 594520064 bytes 2023-12-16T11:23:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 653072384 bytes 2023-12-16T11:23:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 682785792 bytes 2023-12-16T11:23:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 751364096 bytes 2023-12-16T11:23:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 803577856 bytes 2023-12-16T11:23:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 810054656 bytes 2023-12-16T11:23:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 838588416 bytes 2023-12-16T11:23:57.757 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:23:57.753+0000 7f13f7b15640 -1 reset not still connected to 0x5589c174b450 2023-12-16T11:23:57.758 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:23:57.754+0000 7f13f7b15640 -1 reset not still connected to 0x5589c75a1040 2023-12-16T11:23:57.758 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:23:57.754+0000 7f13f7b15640 -1 reset not still connected to 0x5589c75a1a00 2023-12-16T11:23:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 922196992 bytes 2023-12-16T11:23:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 961671168 bytes 2023-12-16T11:23:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1016343552 bytes 2023-12-16T11:23:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1081140224 bytes 2023-12-16T11:23:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1101049856 bytes 2023-12-16T11:23:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1101671424 bytes 2023-12-16T11:23:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1162735616 bytes 2023-12-16T11:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1163331584 bytes 2023-12-16T11:23:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1212686336 bytes 2023-12-16T11:23:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1266274304 bytes 2023-12-16T11:23:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1287577600 bytes 2023-12-16T11:23:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1366027264 bytes 2023-12-16T11:23:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1380301824 bytes 2023-12-16T11:24:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1455723520 bytes 2023-12-16T11:24:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1529511936 bytes 2023-12-16T11:24:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1551610880 bytes 2023-12-16T11:24:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1639649280 bytes 2023-12-16T11:24:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1683481600 bytes 2023-12-16T11:24:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1697473536 bytes 2023-12-16T11:24:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1727301632 bytes 2023-12-16T11:24:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1742312448 bytes 2023-12-16T11:24:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1826990080 bytes 2023-12-16T11:24:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1850926080 bytes 2023-12-16T11:24:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1915681792 bytes 2023-12-16T11:24:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1974945792 bytes 2023-12-16T11:24:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1975545856 bytes 2023-12-16T11:24:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2027338752 bytes 2023-12-16T11:24:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2096859136 bytes 2023-12-16T11:24:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2124467200 bytes 2023-12-16T11:24:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2193783808 bytes 2023-12-16T11:24:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2239824896 bytes 2023-12-16T11:24:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2255148032 bytes 2023-12-16T11:24:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2276808704 bytes 2023-12-16T11:24:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2376361984 bytes 2023-12-16T11:24:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2399787008 bytes 2023-12-16T11:24:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2422136832 bytes 2023-12-16T11:24:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2489340928 bytes 2023-12-16T11:24:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2531281920 bytes 2023-12-16T11:24:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2560381952 bytes 2023-12-16T11:24:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2587987968 bytes 2023-12-16T11:24:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2590108672 bytes 2023-12-16T11:24:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2605614080 bytes 2023-12-16T11:24:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2611557376 bytes 2023-12-16T11:24:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2694931456 bytes 2023-12-16T11:24:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2731868160 bytes 2023-12-16T11:24:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2795892736 bytes 2023-12-16T11:24:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2834446336 bytes 2023-12-16T11:24:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2836976640 bytes 2023-12-16T11:24:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2874539008 bytes 2023-12-16T11:24:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 3002893312 bytes 2023-12-16T11:24:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 3073088512 bytes 2023-12-16T11:24:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 3159488512 bytes 2023-12-16T11:24:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 3169035264 bytes 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_21 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:07.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:24:07.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:24:07.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:24:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:24:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-16T11:24:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:24:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_21': 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 135c4c94061e 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.135c4c94061e 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:07 2023 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:07 2023 2023-12-16T11:24:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:07 2023 2023-12-16T11:24:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:24:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:24:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 701a286f-a667-4073-9a30-e7a68ef93705 2023-12-16T11:24:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:24:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:24:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-16T11:24:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:24:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:24:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:24:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:24:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:24:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 13364224 bytes 2023-12-16T11:24:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 42266624 bytes 2023-12-16T11:24:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 59801600 bytes 2023-12-16T11:24:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 107078656 bytes 2023-12-16T11:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 159887360 bytes 2023-12-16T11:24:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 173513728 bytes 2023-12-16T11:24:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 175314944 bytes 2023-12-16T11:24:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 230705152 bytes 2023-12-16T11:24:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 269761536 bytes 2023-12-16T11:24:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 315704320 bytes 2023-12-16T11:24:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 384254976 bytes 2023-12-16T11:24:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 492906496 bytes 2023-12-16T11:24:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 521261056 bytes 2023-12-16T11:24:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 521848832 bytes 2023-12-16T11:24:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 538542080 bytes 2023-12-16T11:24:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 577363968 bytes 2023-12-16T11:24:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 615835648 bytes 2023-12-16T11:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 672667648 bytes 2023-12-16T11:24:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 685596672 bytes 2023-12-16T11:24:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 772532224 bytes 2023-12-16T11:24:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 798693376 bytes 2023-12-16T11:24:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 819636224 bytes 2023-12-16T11:24:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 830850048 bytes 2023-12-16T11:24:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 839677952 bytes 2023-12-16T11:24:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 882279424 bytes 2023-12-16T11:24:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 944949248 bytes 2023-12-16T11:24:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1000449024 bytes 2023-12-16T11:24:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1042353152 bytes 2023-12-16T11:24:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1061313536 bytes 2023-12-16T11:24:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1089057792 bytes 2023-12-16T11:24:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_22 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:24:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-16T11:24:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:24:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-16T11:24:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_22': 2023-12-16T11:24:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:24:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:24:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:24:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1368a5cc30f9 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1368a5cc30f9 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:13 2023 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:13 2023 2023-12-16T11:24:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:13 2023 2023-12-16T11:24:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:24:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:24:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: de08cde6-faea-4649-8611-8d30022bde8d 2023-12-16T11:24:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:24:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:24:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-16T11:24:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:24:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:24:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=10 2023-12-16T11:24:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:24:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:24:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 53398528 bytes 2023-12-16T11:24:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 127483904 bytes 2023-12-16T11:24:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 171175936 bytes 2023-12-16T11:24:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 217222144 bytes 2023-12-16T11:24:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 283291648 bytes 2023-12-16T11:24:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 394388480 bytes 2023-12-16T11:24:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 480481280 bytes 2023-12-16T11:24:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 481085440 bytes 2023-12-16T11:24:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 562732032 bytes 2023-12-16T11:24:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 629307392 bytes 2023-12-16T11:24:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 667516928 bytes 2023-12-16T11:24:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 715534336 bytes 2023-12-16T11:24:17.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 774275072 bytes 2023-12-16T11:24:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 830131200 bytes 2023-12-16T11:24:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 900489216 bytes 2023-12-16T11:24:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 930795520 bytes 2023-12-16T11:24:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 965782528 bytes 2023-12-16T11:24:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 1040817152 bytes 2023-12-16T11:24:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 1043807232 bytes 2023-12-16T11:24:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1110069248 bytes 2023-12-16T11:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1123636224 bytes 2023-12-16T11:24:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1154635776 bytes 2023-12-16T11:24:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1191259136 bytes 2023-12-16T11:24:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1222470656 bytes 2023-12-16T11:24:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1251040256 bytes 2023-12-16T11:24:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1280637952 bytes 2023-12-16T11:24:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1360328704 bytes 2023-12-16T11:24:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1381568512 bytes 2023-12-16T11:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1399656448 bytes 2023-12-16T11:24:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1431828480 bytes 2023-12-16T11:24:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1464516608 bytes 2023-12-16T11:24:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1475036160 bytes 2023-12-16T11:24:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1532409856 bytes 2023-12-16T11:24:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1562807296 bytes 2023-12-16T11:24:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1641959424 bytes 2023-12-16T11:24:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1668731904 bytes 2023-12-16T11:24:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1712625664 bytes 2023-12-16T11:24:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1747275776 bytes 2023-12-16T11:24:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1807537152 bytes 2023-12-16T11:24:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1826309120 bytes 2023-12-16T11:24:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1868695552 bytes 2023-12-16T11:24:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1957900288 bytes 2023-12-16T11:24:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1987044352 bytes 2023-12-16T11:24:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2081994752 bytes 2023-12-16T11:24:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2109521920 bytes 2023-12-16T11:24:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2141860864 bytes 2023-12-16T11:24:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2250851328 bytes 2023-12-16T11:24:23.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2280288256 bytes 2023-12-16T11:24:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2332520448 bytes 2023-12-16T11:24:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2365263872 bytes 2023-12-16T11:24:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2396608512 bytes 2023-12-16T11:24:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2435912704 bytes 2023-12-16T11:24:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:24:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:24:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:24:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:24:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_23 2023-12-16T11:24:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:24:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:24:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:24:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:24:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2023-12-16T11:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-16T11:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-16T11:24:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:24:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-16T11:24:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_23': 2023-12-16T11:24:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 137426bc2f72 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.137426bc2f72 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:24 2023 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:24 2023 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:24 2023 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 0ff47fdb-18fc-451e-9837-b80cef74a025 2023-12-16T11:24:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:24:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:24:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-16T11:24:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:24:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:24:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=10 2023-12-16T11:24:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:24:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:24:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 630784 bytes 2023-12-16T11:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 50472960 bytes 2023-12-16T11:24:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 109233152 bytes 2023-12-16T11:24:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 133916672 bytes 2023-12-16T11:24:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 197832704 bytes 2023-12-16T11:24:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 228267008 bytes 2023-12-16T11:24:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 249422848 bytes 2023-12-16T11:24:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 293826560 bytes 2023-12-16T11:24:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 316091392 bytes 2023-12-16T11:24:26.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 348542976 bytes 2023-12-16T11:24:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 366477312 bytes 2023-12-16T11:24:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 381589504 bytes 2023-12-16T11:24:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 416973824 bytes 2023-12-16T11:24:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 489519104 bytes 2023-12-16T11:24:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 528281600 bytes 2023-12-16T11:24:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 566140928 bytes 2023-12-16T11:24:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 611031040 bytes 2023-12-16T11:24:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 623389696 bytes 2023-12-16T11:24:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 673155072 bytes 2023-12-16T11:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 721884160 bytes 2023-12-16T11:24:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 722498560 bytes 2023-12-16T11:24:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 729058304 bytes 2023-12-16T11:24:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 742360064 bytes 2023-12-16T11:24:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 777915392 bytes 2023-12-16T11:24:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 787931136 bytes 2023-12-16T11:24:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 866382848 bytes 2023-12-16T11:24:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 871698432 bytes 2023-12-16T11:24:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 922418176 bytes 2023-12-16T11:24:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 923022336 bytes 2023-12-16T11:24:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 957915136 bytes 2023-12-16T11:24:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 981069824 bytes 2023-12-16T11:24:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1017726976 bytes 2023-12-16T11:24:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1106597888 bytes 2023-12-16T11:24:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1144594432 bytes 2023-12-16T11:24:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1195456512 bytes 2023-12-16T11:24:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1231042560 bytes 2023-12-16T11:24:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1271372800 bytes 2023-12-16T11:24:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1341960192 bytes 2023-12-16T11:24:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1400830976 bytes 2023-12-16T11:24:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1431797760 bytes 2023-12-16T11:24:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1504037888 bytes 2023-12-16T11:24:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1554398208 bytes 2023-12-16T11:24:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1602151424 bytes 2023-12-16T11:24:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1602776064 bytes 2023-12-16T11:24:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1692815360 bytes 2023-12-16T11:24:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1730591744 bytes 2023-12-16T11:24:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1739138048 bytes 2023-12-16T11:24:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1836171264 bytes 2023-12-16T11:24:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1869949952 bytes 2023-12-16T11:24:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1870575616 bytes 2023-12-16T11:24:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1914128384 bytes 2023-12-16T11:24:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 1939012608 bytes 2023-12-16T11:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 1974443008 bytes 2023-12-16T11:24:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2051046400 bytes 2023-12-16T11:24:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2129156096 bytes 2023-12-16T11:24:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2156377088 bytes 2023-12-16T11:24:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2207694848 bytes 2023-12-16T11:24:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2245038080 bytes 2023-12-16T11:24:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2288802816 bytes 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_24 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-16T11:24:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:24:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:24:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:24:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:24:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-16T11:24:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:24:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_24': 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1380ae805d4a 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1380ae805d4a 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:24:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:24:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:35 2023 2023-12-16T11:24:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:35 2023 2023-12-16T11:24:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:35 2023 2023-12-16T11:24:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:24:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:24:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: baa6790f-87a2-4cff-9c34-1c577908fc53 2023-12-16T11:24:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:24:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:24:36.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-16T11:24:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:24:36.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:24:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:24:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:24:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:24:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 82479104 bytes 2023-12-16T11:24:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 130070528 bytes 2023-12-16T11:24:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 159877120 bytes 2023-12-16T11:24:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 220735488 bytes 2023-12-16T11:24:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 344885248 bytes 2023-12-16T11:24:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 366187520 bytes 2023-12-16T11:24:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 386893824 bytes 2023-12-16T11:24:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 485357568 bytes 2023-12-16T11:24:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 492339200 bytes 2023-12-16T11:24:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 542668800 bytes 2023-12-16T11:24:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 601001984 bytes 2023-12-16T11:24:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 605506560 bytes 2023-12-16T11:24:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 624983040 bytes 2023-12-16T11:24:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 702840832 bytes 2023-12-16T11:24:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 704104448 bytes 2023-12-16T11:24:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 742145024 bytes 2023-12-16T11:24:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 770318336 bytes 2023-12-16T11:24:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 794997760 bytes 2023-12-16T11:24:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 860414976 bytes 2023-12-16T11:24:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 890654720 bytes 2023-12-16T11:24:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 936402944 bytes 2023-12-16T11:24:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 961568768 bytes 2023-12-16T11:24:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 979068928 bytes 2023-12-16T11:24:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1013029888 bytes 2023-12-16T11:24:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1081269248 bytes 2023-12-16T11:24:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1139230720 bytes 2023-12-16T11:24:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1165976576 bytes 2023-12-16T11:24:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1245303808 bytes 2023-12-16T11:24:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1291950080 bytes 2023-12-16T11:24:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1317005312 bytes 2023-12-16T11:24:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1359011840 bytes 2023-12-16T11:24:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1403589632 bytes 2023-12-16T11:24:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1435964416 bytes 2023-12-16T11:24:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1491195904 bytes 2023-12-16T11:24:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1517895680 bytes 2023-12-16T11:24:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1581060096 bytes 2023-12-16T11:24:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1617608704 bytes 2023-12-16T11:24:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1711376384 bytes 2023-12-16T11:24:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:24:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:24:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:24:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_25 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2023-12-16T11:24:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:24:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:24:43.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-16T11:24:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:24:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-16T11:24:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:24:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_25': 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 138bfc606bce 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.138bfc606bce 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:24:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:24:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:43 2023 2023-12-16T11:24:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:43 2023 2023-12-16T11:24:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:43 2023 2023-12-16T11:24:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:24:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:24:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 6ff4a59e-4940-43b0-92d3-556cb67e8005 2023-12-16T11:24:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:24:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:24:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-16T11:24:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:24:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:24:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:24:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:24:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:24:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 8762368 bytes 2023-12-16T11:24:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 42038272 bytes 2023-12-16T11:24:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 94546944 bytes 2023-12-16T11:24:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 155581440 bytes 2023-12-16T11:24:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 204000256 bytes 2023-12-16T11:24:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 276783104 bytes 2023-12-16T11:24:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 312103936 bytes 2023-12-16T11:24:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 312702976 bytes 2023-12-16T11:24:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 344213504 bytes 2023-12-16T11:24:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 399675392 bytes 2023-12-16T11:24:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 425710592 bytes 2023-12-16T11:24:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 438247424 bytes 2023-12-16T11:24:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 504820736 bytes 2023-12-16T11:24:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 523821056 bytes 2023-12-16T11:24:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 569849856 bytes 2023-12-16T11:24:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 629326848 bytes 2023-12-16T11:24:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 646097920 bytes 2023-12-16T11:24:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 679482368 bytes 2023-12-16T11:24:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 721680384 bytes 2023-12-16T11:24:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 757416960 bytes 2023-12-16T11:24:47.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 815071232 bytes 2023-12-16T11:24:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 848090112 bytes 2023-12-16T11:24:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 890452992 bytes 2023-12-16T11:24:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 920902656 bytes 2023-12-16T11:24:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 930866176 bytes 2023-12-16T11:24:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 953220096 bytes 2023-12-16T11:24:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 974680064 bytes 2023-12-16T11:24:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1045282816 bytes 2023-12-16T11:24:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1082352640 bytes 2023-12-16T11:24:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1157899264 bytes 2023-12-16T11:24:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1219288064 bytes 2023-12-16T11:24:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1220071424 bytes 2023-12-16T11:24:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1240193024 bytes 2023-12-16T11:24:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1298236416 bytes 2023-12-16T11:24:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1318333440 bytes 2023-12-16T11:24:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1395973120 bytes 2023-12-16T11:24:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1424638976 bytes 2023-12-16T11:24:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1470373888 bytes 2023-12-16T11:24:50.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1522251776 bytes 2023-12-16T11:24:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1572833280 bytes 2023-12-16T11:24:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1597067264 bytes 2023-12-16T11:24:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1681864704 bytes 2023-12-16T11:24:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1700464640 bytes 2023-12-16T11:24:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1768837120 bytes 2023-12-16T11:24:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1775448064 bytes 2023-12-16T11:24:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1793220608 bytes 2023-12-16T11:24:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1818021888 bytes 2023-12-16T11:24:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1863918592 bytes 2023-12-16T11:24:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1890991104 bytes 2023-12-16T11:24:52.527 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4285085425676891) > weight (0.25) 2023-12-16T11:24:52.528 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.31747780250890145) > weight (0.25) 2023-12-16T11:24:52.528 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9941164021633966) > weight (0.25) 2023-12-16T11:24:52.528 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5699795650375016) > weight (0.25) 2023-12-16T11:24:52.528 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-16T11:24:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1918163968 bytes 2023-12-16T11:24:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1996741632 bytes 2023-12-16T11:24:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2007084032 bytes 2023-12-16T11:24:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2068428800 bytes 2023-12-16T11:24:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2091223040 bytes 2023-12-16T11:24:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2131094528 bytes 2023-12-16T11:24:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2168683520 bytes 2023-12-16T11:24:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2225907712 bytes 2023-12-16T11:24:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2245242880 bytes 2023-12-16T11:24:54.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2278599680 bytes 2023-12-16T11:24:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2297926656 bytes 2023-12-16T11:24:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2344372224 bytes 2023-12-16T11:24:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2361279488 bytes 2023-12-16T11:24:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2385973248 bytes 2023-12-16T11:24:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2439382016 bytes 2023-12-16T11:24:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2456676352 bytes 2023-12-16T11:24:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2502816768 bytes 2023-12-16T11:24:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2575525888 bytes 2023-12-16T11:24:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2596763648 bytes 2023-12-16T11:24:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2597365760 bytes 2023-12-16T11:24:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2698991616 bytes 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_26 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:24:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:24:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:24:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2023-12-16T11:24:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:24:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:24:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:24:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-16T11:24:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:56.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:24:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:24:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-16T11:24:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:24:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-16T11:24:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_26': 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1397d34aa8d6 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1397d34aa8d6 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:56 2023 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:56 2023 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:56 2023 2023-12-16T11:24:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:24:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:24:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c10d0adc-6101-4c60-acef-48112957bc4c 2023-12-16T11:24:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-16T11:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:24:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:24:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=13 2023-12-16T11:24:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:24:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:24:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 10664960 bytes 2023-12-16T11:24:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 33357824 bytes 2023-12-16T11:24:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 47440896 bytes 2023-12-16T11:24:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 58640384 bytes 2023-12-16T11:24:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 126059520 bytes 2023-12-16T11:24:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 161895424 bytes 2023-12-16T11:24:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 198631424 bytes 2023-12-16T11:24:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 210806784 bytes 2023-12-16T11:24:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 260001792 bytes 2023-12-16T11:24:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 277625856 bytes 2023-12-16T11:24:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 320454656 bytes 2023-12-16T11:24:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 375566336 bytes 2023-12-16T11:24:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 438505472 bytes 2023-12-16T11:24:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 460302336 bytes 2023-12-16T11:24:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 493047808 bytes 2023-12-16T11:24:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 504940544 bytes 2023-12-16T11:24:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 530525184 bytes 2023-12-16T11:24:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 595253248 bytes 2023-12-16T11:25:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 683352064 bytes 2023-12-16T11:25:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 749227008 bytes 2023-12-16T11:25:00.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 802409472 bytes 2023-12-16T11:25:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 826966016 bytes 2023-12-16T11:25:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 909732864 bytes 2023-12-16T11:25:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 954064896 bytes 2023-12-16T11:25:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1009285120 bytes 2023-12-16T11:25:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1022143488 bytes 2023-12-16T11:25:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1025715200 bytes 2023-12-16T11:25:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1079571456 bytes 2023-12-16T11:25:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1110022144 bytes 2023-12-16T11:25:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1138095104 bytes 2023-12-16T11:25:02.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1180526592 bytes 2023-12-16T11:25:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1225441280 bytes 2023-12-16T11:25:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1280155648 bytes 2023-12-16T11:25:03.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1333095424 bytes 2023-12-16T11:25:03.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1333747712 bytes 2023-12-16T11:25:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1355146240 bytes 2023-12-16T11:25:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1385844736 bytes 2023-12-16T11:25:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1429945344 bytes 2023-12-16T11:25:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1491968000 bytes 2023-12-16T11:25:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1503695872 bytes 2023-12-16T11:25:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1530587136 bytes 2023-12-16T11:25:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1581170688 bytes 2023-12-16T11:25:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1646827520 bytes 2023-12-16T11:25:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1703709696 bytes 2023-12-16T11:25:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1717446656 bytes 2023-12-16T11:25:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1820115968 bytes 2023-12-16T11:25:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1888172032 bytes 2023-12-16T11:25:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1923773440 bytes 2023-12-16T11:25:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1961017344 bytes 2023-12-16T11:25:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1961620480 bytes 2023-12-16T11:25:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2000822272 bytes 2023-12-16T11:25:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2071630848 bytes 2023-12-16T11:25:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2132225024 bytes 2023-12-16T11:25:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2172596224 bytes 2023-12-16T11:25:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2182972416 bytes 2023-12-16T11:25:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2237989888 bytes 2023-12-16T11:25:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2272479232 bytes 2023-12-16T11:25:07.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2293136384 bytes 2023-12-16T11:25:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2317231104 bytes 2023-12-16T11:25:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2359533568 bytes 2023-12-16T11:25:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2446171136 bytes 2023-12-16T11:25:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2483132416 bytes 2023-12-16T11:25:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2504728576 bytes 2023-12-16T11:25:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2544854016 bytes 2023-12-16T11:25:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2545481728 bytes 2023-12-16T11:25:08.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2564308992 bytes 2023-12-16T11:25:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2660682752 bytes 2023-12-16T11:25:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2706927616 bytes 2023-12-16T11:25:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2755044352 bytes 2023-12-16T11:25:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2805324800 bytes 2023-12-16T11:25:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 2908016640 bytes 2023-12-16T11:25:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 2975322112 bytes 2023-12-16T11:25:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 3029284864 bytes 2023-12-16T11:25:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:25:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:25:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:25:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:25:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_27 2023-12-16T11:25:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:25:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:25:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:25:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:25:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2023-12-16T11:25:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:25:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:25:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-16T11:25:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:25:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-16T11:25:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:25:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-16T11:25:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_27': 2023-12-16T11:25:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:25:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:25:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:25:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13a3eed35ed5 2023-12-16T11:25:11.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13a3eed35ed5 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:10 2023 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:10 2023 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:10 2023 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 5da77513-de12-45d4-b866-435a3db39e9a 2023-12-16T11:25:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:25:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:25:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-16T11:25:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:25:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:25:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=13 2023-12-16T11:25:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:25:11.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:25:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 19576832 bytes 2023-12-16T11:25:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 50934784 bytes 2023-12-16T11:25:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 72097792 bytes 2023-12-16T11:25:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 87808000 bytes 2023-12-16T11:25:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 138595328 bytes 2023-12-16T11:25:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 242341888 bytes 2023-12-16T11:25:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 283183104 bytes 2023-12-16T11:25:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 361815040 bytes 2023-12-16T11:25:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 439393280 bytes 2023-12-16T11:25:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 503058432 bytes 2023-12-16T11:25:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 537078784 bytes 2023-12-16T11:25:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 559497216 bytes 2023-12-16T11:25:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 644615168 bytes 2023-12-16T11:25:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 673899520 bytes 2023-12-16T11:25:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 702965760 bytes 2023-12-16T11:25:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 723512320 bytes 2023-12-16T11:25:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 724119552 bytes 2023-12-16T11:25:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 768363520 bytes 2023-12-16T11:25:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 789928960 bytes 2023-12-16T11:25:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 829831168 bytes 2023-12-16T11:25:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 887600128 bytes 2023-12-16T11:25:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 941543424 bytes 2023-12-16T11:25:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 993261568 bytes 2023-12-16T11:25:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1056947200 bytes 2023-12-16T11:25:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1093373952 bytes 2023-12-16T11:25:15.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5463619429538581) > weight (0.25) 2023-12-16T11:25:15.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9768593481806235) > weight (0.25) 2023-12-16T11:25:15.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4230148430515206) > weight (0.25) 2023-12-16T11:25:15.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8258694075223517) > weight (0.25) 2023-12-16T11:25:15.541 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-12-16T11:25:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1105967104 bytes 2023-12-16T11:25:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1205578752 bytes 2023-12-16T11:25:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1250597888 bytes 2023-12-16T11:25:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1313900544 bytes 2023-12-16T11:25:16.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1357053952 bytes 2023-12-16T11:25:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1396590592 bytes 2023-12-16T11:25:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1446344704 bytes 2023-12-16T11:25:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1498651648 bytes 2023-12-16T11:25:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1575534592 bytes 2023-12-16T11:25:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1662971904 bytes 2023-12-16T11:25:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1663586304 bytes 2023-12-16T11:25:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1681022976 bytes 2023-12-16T11:25:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1752053760 bytes 2023-12-16T11:25:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1807752192 bytes 2023-12-16T11:25:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1808380928 bytes 2023-12-16T11:25:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1831286784 bytes 2023-12-16T11:25:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1861271552 bytes 2023-12-16T11:25:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1945842688 bytes 2023-12-16T11:25:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1950669824 bytes 2023-12-16T11:25:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1999605760 bytes 2023-12-16T11:25:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2064795648 bytes 2023-12-16T11:25:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2107893760 bytes 2023-12-16T11:25:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2120926208 bytes 2023-12-16T11:25:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2218370048 bytes 2023-12-16T11:25:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2280415232 bytes 2023-12-16T11:25:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2371524608 bytes 2023-12-16T11:25:20.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2419080192 bytes 2023-12-16T11:25:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2469880832 bytes 2023-12-16T11:25:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2500758528 bytes 2023-12-16T11:25:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2543494144 bytes 2023-12-16T11:25:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2596597760 bytes 2023-12-16T11:25:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2618498048 bytes 2023-12-16T11:25:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2642608128 bytes 2023-12-16T11:25:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2690113536 bytes 2023-12-16T11:25:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2730618880 bytes 2023-12-16T11:25:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2756354048 bytes 2023-12-16T11:25:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2858600448 bytes 2023-12-16T11:25:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2892894208 bytes 2023-12-16T11:25:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2941655040 bytes 2023-12-16T11:25:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 3015259136 bytes 2023-12-16T11:25:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 3117486080 bytes 2023-12-16T11:25:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 3148735488 bytes 2023-12-16T11:25:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 3173648384 bytes 2023-12-16T11:25:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 3237008384 bytes 2023-12-16T11:25:23.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 3240657920 bytes 2023-12-16T11:25:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:25:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:25:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:25:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:25:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_28 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:25:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:25:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:25:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:25:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2023-12-16T11:25:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:25:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:25:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-16T11:25:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:25:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-16T11:25:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:25:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_28': 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13af8b626e28 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13af8b626e28 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:25:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:24 2023 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:24 2023 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:24 2023 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 89b02294-084e-47fc-a8c6-56a13899278d 2023-12-16T11:25:25.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:25:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:25:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-16T11:25:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:25:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:25:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=9 2023-12-16T11:25:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:25:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:25:25.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 29380608 bytes 2023-12-16T11:25:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 57483264 bytes 2023-12-16T11:25:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 122207232 bytes 2023-12-16T11:25:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 153748480 bytes 2023-12-16T11:25:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 154369024 bytes 2023-12-16T11:25:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 198247424 bytes 2023-12-16T11:25:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 239130624 bytes 2023-12-16T11:25:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 279902208 bytes 2023-12-16T11:25:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 378561536 bytes 2023-12-16T11:25:26.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 414799872 bytes 2023-12-16T11:25:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 473035776 bytes 2023-12-16T11:25:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 519003136 bytes 2023-12-16T11:25:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 555960320 bytes 2023-12-16T11:25:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 619540480 bytes 2023-12-16T11:25:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 678765568 bytes 2023-12-16T11:25:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 750064640 bytes 2023-12-16T11:25:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 794630144 bytes 2023-12-16T11:25:28.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 811812864 bytes 2023-12-16T11:25:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 823955456 bytes 2023-12-16T11:25:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 869040128 bytes 2023-12-16T11:25:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 907501568 bytes 2023-12-16T11:25:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 965238784 bytes 2023-12-16T11:25:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1025137664 bytes 2023-12-16T11:25:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1025748992 bytes 2023-12-16T11:25:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1073116160 bytes 2023-12-16T11:25:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1098901504 bytes 2023-12-16T11:25:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1099510784 bytes 2023-12-16T11:25:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1130420224 bytes 2023-12-16T11:25:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1153983488 bytes 2023-12-16T11:25:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1206908928 bytes 2023-12-16T11:25:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1269903360 bytes 2023-12-16T11:25:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1298298880 bytes 2023-12-16T11:25:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1410004992 bytes 2023-12-16T11:25:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1467130880 bytes 2023-12-16T11:25:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1480324096 bytes 2023-12-16T11:25:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1534242816 bytes 2023-12-16T11:25:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1550796800 bytes 2023-12-16T11:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1554794496 bytes 2023-12-16T11:25:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1566219264 bytes 2023-12-16T11:25:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1638088704 bytes 2023-12-16T11:25:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1677704192 bytes 2023-12-16T11:25:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1772172288 bytes 2023-12-16T11:25:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1823201280 bytes 2023-12-16T11:25:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1878028288 bytes 2023-12-16T11:25:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1878656000 bytes 2023-12-16T11:25:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1901480960 bytes 2023-12-16T11:25:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1940047872 bytes 2023-12-16T11:25:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1959657472 bytes 2023-12-16T11:25:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2008020992 bytes 2023-12-16T11:25:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2020217856 bytes 2023-12-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_29 2023-12-16T11:25:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:25:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:34.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-16T11:25:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:25:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-16T11:25:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_29': 2023-12-16T11:25:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13bb64af7412 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13bb64af7412 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:34 2023 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:34 2023 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:34 2023 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: fafbb98e-d219-40d4-93e6-5c734a4ed51d 2023-12-16T11:25:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-16T11:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:25:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:25:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:25:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 67853312 bytes 2023-12-16T11:25:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 119583744 bytes 2023-12-16T11:25:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 173004800 bytes 2023-12-16T11:25:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 227081216 bytes 2023-12-16T11:25:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 259871744 bytes 2023-12-16T11:25:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 265037824 bytes 2023-12-16T11:25:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 299288576 bytes 2023-12-16T11:25:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 314822656 bytes 2023-12-16T11:25:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 377348096 bytes 2023-12-16T11:25:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 480230400 bytes 2023-12-16T11:25:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 540237824 bytes 2023-12-16T11:25:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 588380160 bytes 2023-12-16T11:25:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 637333504 bytes 2023-12-16T11:25:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 668008448 bytes 2023-12-16T11:25:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 732617728 bytes 2023-12-16T11:25:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 826960896 bytes 2023-12-16T11:25:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 875872256 bytes 2023-12-16T11:25:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 918310912 bytes 2023-12-16T11:25:38.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 931333120 bytes 2023-12-16T11:25:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 949626880 bytes 2023-12-16T11:25:39.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 995062784 bytes 2023-12-16T11:25:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 995678208 bytes 2023-12-16T11:25:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1023301632 bytes 2023-12-16T11:25:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1063547904 bytes 2023-12-16T11:25:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1068203008 bytes 2023-12-16T11:25:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1141827584 bytes 2023-12-16T11:25:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1211836416 bytes 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_30 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-16T11:25:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:25:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:25:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:25:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:25:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-16T11:25:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:25:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-16T11:25:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_30': 2023-12-16T11:25:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:25:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13c73ee39ec8 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13c73ee39ec8 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:40 2023 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:40 2023 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:40 2023 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 42e05876-1774-4fdf-a810-7063093f5060 2023-12-16T11:25:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:25:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:25:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-16T11:25:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:25:41.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:25:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:25:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:25:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:25:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 27052032 bytes 2023-12-16T11:25:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 47791104 bytes 2023-12-16T11:25:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 106782720 bytes 2023-12-16T11:25:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 157977600 bytes 2023-12-16T11:25:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 194653184 bytes 2023-12-16T11:25:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 229866496 bytes 2023-12-16T11:25:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 230495232 bytes 2023-12-16T11:25:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 274466816 bytes 2023-12-16T11:25:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 330472448 bytes 2023-12-16T11:25:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 393693184 bytes 2023-12-16T11:25:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 399297536 bytes 2023-12-16T11:25:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 422749184 bytes 2023-12-16T11:25:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 441735168 bytes 2023-12-16T11:25:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 459054080 bytes 2023-12-16T11:25:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 498891776 bytes 2023-12-16T11:25:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 567736320 bytes 2023-12-16T11:25:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 590958592 bytes 2023-12-16T11:25:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 620103680 bytes 2023-12-16T11:25:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 686759936 bytes 2023-12-16T11:25:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 739350528 bytes 2023-12-16T11:25:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 767349760 bytes 2023-12-16T11:25:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 768210944 bytes 2023-12-16T11:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 812252160 bytes 2023-12-16T11:25:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 833384448 bytes 2023-12-16T11:25:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 865257472 bytes 2023-12-16T11:25:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 919611392 bytes 2023-12-16T11:25:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 940169216 bytes 2023-12-16T11:25:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 966870016 bytes 2023-12-16T11:25:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1030087680 bytes 2023-12-16T11:25:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1117111296 bytes 2023-12-16T11:25:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1191018496 bytes 2023-12-16T11:25:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1225586688 bytes 2023-12-16T11:25:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1229458432 bytes 2023-12-16T11:25:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1242917888 bytes 2023-12-16T11:25:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1335743488 bytes 2023-12-16T11:25:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1410997248 bytes 2023-12-16T11:25:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1434429440 bytes 2023-12-16T11:25:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1475646464 bytes 2023-12-16T11:25:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1518473216 bytes 2023-12-16T11:25:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:25:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:25:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_31 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-16T11:25:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:25:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-16T11:25:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:25:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_31': 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13d362c32daa 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13d362c32daa 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:25:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:48 2023 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:48 2023 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:48 2023 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c20fd292-e9fd-4021-a507-e180446b4cb3 2023-12-16T11:25:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:25:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:25:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-16T11:25:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:25:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:25:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:25:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:25:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:25:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 87615488 bytes 2023-12-16T11:25:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 131932160 bytes 2023-12-16T11:25:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 181706752 bytes 2023-12-16T11:25:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 211170304 bytes 2023-12-16T11:25:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 241933312 bytes 2023-12-16T11:25:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 290879488 bytes 2023-12-16T11:25:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 368189440 bytes 2023-12-16T11:25:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 405005312 bytes 2023-12-16T11:25:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 413120512 bytes 2023-12-16T11:25:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 479804416 bytes 2023-12-16T11:25:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 545561600 bytes 2023-12-16T11:25:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 583615488 bytes 2023-12-16T11:25:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 608793600 bytes 2023-12-16T11:25:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 619808768 bytes 2023-12-16T11:25:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 707667968 bytes 2023-12-16T11:25:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 727737344 bytes 2023-12-16T11:25:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 809120768 bytes 2023-12-16T11:25:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 859314176 bytes 2023-12-16T11:25:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 905488384 bytes 2023-12-16T11:25:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 939916288 bytes 2023-12-16T11:25:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 997467136 bytes 2023-12-16T11:25:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1025476608 bytes 2023-12-16T11:25:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1075718144 bytes 2023-12-16T11:25:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1144229888 bytes 2023-12-16T11:25:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1270584320 bytes 2023-12-16T11:25:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:25:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:25:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:25:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:25:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_32 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:25:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:25:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:25:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:25:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:25:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:25:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-16T11:25:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:25:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_32': 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13dff632b581 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13dff632b581 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:25:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:53 2023 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:53 2023 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:53 2023 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 5d737155-c20e-4e60-ba8a-f1c27e0ded2f 2023-12-16T11:25:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:25:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:25:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-16T11:25:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:25:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:25:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:25:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:25:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:25:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:25:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:25:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 21893120 bytes 2023-12-16T11:25:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 95166464 bytes 2023-12-16T11:25:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 146305024 bytes 2023-12-16T11:25:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 257841152 bytes 2023-12-16T11:25:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 294175744 bytes 2023-12-16T11:25:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 351125504 bytes 2023-12-16T11:25:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 376844288 bytes 2023-12-16T11:25:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 452945920 bytes 2023-12-16T11:25:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 457639936 bytes 2023-12-16T11:25:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 470085632 bytes 2023-12-16T11:25:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 507944960 bytes 2023-12-16T11:25:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 536751104 bytes 2023-12-16T11:25:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 562039808 bytes 2023-12-16T11:25:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 598565888 bytes 2023-12-16T11:25:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 599138304 bytes 2023-12-16T11:25:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 686299136 bytes 2023-12-16T11:25:58.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 760157184 bytes 2023-12-16T11:25:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 771818496 bytes 2023-12-16T11:25:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 843043840 bytes 2023-12-16T11:25:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 926366720 bytes 2023-12-16T11:25:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 998890496 bytes 2023-12-16T11:25:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1011357696 bytes 2023-12-16T11:25:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1050673152 bytes 2023-12-16T11:25:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1098499072 bytes 2023-12-16T11:25:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1162981376 bytes 2023-12-16T11:25:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1208629248 bytes 2023-12-16T11:26:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1303567360 bytes 2023-12-16T11:26:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1316694016 bytes 2023-12-16T11:26:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1428207616 bytes 2023-12-16T11:26:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1488953344 bytes 2023-12-16T11:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1523163136 bytes 2023-12-16T11:26:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1550049280 bytes 2023-12-16T11:26:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1611265024 bytes 2023-12-16T11:26:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1631040512 bytes 2023-12-16T11:26:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1667223552 bytes 2023-12-16T11:26:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1717238784 bytes 2023-12-16T11:26:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:26:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:26:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:26:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_33 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-16T11:26:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:26:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_33': 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13ebb10bea57 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13ebb10bea57 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:02 2023 2023-12-16T11:26:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:02 2023 2023-12-16T11:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:02 2023 2023-12-16T11:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: eb7a408d-b979-4c72-9d19-d5d246a9c508 2023-12-16T11:26:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:26:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:26:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=14 2023-12-16T11:26:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:26:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:26:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 53506048 bytes 2023-12-16T11:26:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 113532928 bytes 2023-12-16T11:26:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 166793216 bytes 2023-12-16T11:26:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 199708672 bytes 2023-12-16T11:26:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 254828544 bytes 2023-12-16T11:26:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 291956736 bytes 2023-12-16T11:26:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 352819200 bytes 2023-12-16T11:26:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 398948352 bytes 2023-12-16T11:26:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 443940864 bytes 2023-12-16T11:26:05.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 485075968 bytes 2023-12-16T11:26:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 524749824 bytes 2023-12-16T11:26:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 578053120 bytes 2023-12-16T11:26:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 604838912 bytes 2023-12-16T11:26:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 636692480 bytes 2023-12-16T11:26:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 667259904 bytes 2023-12-16T11:26:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 725156864 bytes 2023-12-16T11:26:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 744341504 bytes 2023-12-16T11:26:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 830592000 bytes 2023-12-16T11:26:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 862325760 bytes 2023-12-16T11:26:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 910342144 bytes 2023-12-16T11:26:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 972790784 bytes 2023-12-16T11:26:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1034268672 bytes 2023-12-16T11:26:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1056841728 bytes 2023-12-16T11:26:07.679 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1124667392 bytes 2023-12-16T11:26:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1138627584 bytes 2023-12-16T11:26:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1139228672 bytes 2023-12-16T11:26:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1175424000 bytes 2023-12-16T11:26:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1213791232 bytes 2023-12-16T11:26:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1300592640 bytes 2023-12-16T11:26:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1333978112 bytes 2023-12-16T11:26:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1372810240 bytes 2023-12-16T11:26:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1428177920 bytes 2023-12-16T11:26:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1428789248 bytes 2023-12-16T11:26:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1466811392 bytes 2023-12-16T11:26:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1489304576 bytes 2023-12-16T11:26:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1535917056 bytes 2023-12-16T11:26:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1558855680 bytes 2023-12-16T11:26:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1601040384 bytes 2023-12-16T11:26:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1653974016 bytes 2023-12-16T11:26:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1717463040 bytes 2023-12-16T11:26:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1760376832 bytes 2023-12-16T11:26:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1827356672 bytes 2023-12-16T11:26:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1867083776 bytes 2023-12-16T11:26:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1896059904 bytes 2023-12-16T11:26:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1942692864 bytes 2023-12-16T11:26:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1954979840 bytes 2023-12-16T11:26:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1980502016 bytes 2023-12-16T11:26:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2015155200 bytes 2023-12-16T11:26:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2054083584 bytes 2023-12-16T11:26:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2065844224 bytes 2023-12-16T11:26:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2091354112 bytes 2023-12-16T11:26:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2105282560 bytes 2023-12-16T11:26:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2152005632 bytes 2023-12-16T11:26:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2177214464 bytes 2023-12-16T11:26:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2192717824 bytes 2023-12-16T11:26:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2229814272 bytes 2023-12-16T11:26:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2249797632 bytes 2023-12-16T11:26:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2353659904 bytes 2023-12-16T11:26:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2397885440 bytes 2023-12-16T11:26:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2439510016 bytes 2023-12-16T11:26:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2475364352 bytes 2023-12-16T11:26:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2501367808 bytes 2023-12-16T11:26:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2509199360 bytes 2023-12-16T11:26:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2571132928 bytes 2023-12-16T11:26:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2644894720 bytes 2023-12-16T11:26:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2656870400 bytes 2023-12-16T11:26:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2696323072 bytes 2023-12-16T11:26:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2751962112 bytes 2023-12-16T11:26:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2788925440 bytes 2023-12-16T11:26:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2834549760 bytes 2023-12-16T11:26:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 2847563776 bytes 2023-12-16T11:26:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 2894578688 bytes 2023-12-16T11:26:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 2942952448 bytes 2023-12-16T11:26:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 2957519872 bytes 2023-12-16T11:26:16.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 2986151936 bytes 2023-12-16T11:26:16.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 2994042880 bytes 2023-12-16T11:26:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3000097792 bytes 2023-12-16T11:26:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3044619264 bytes 2023-12-16T11:26:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3097935872 bytes 2023-12-16T11:26:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:8000 IOs, 3123842048 bytes 2023-12-16T11:26:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_34 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-16T11:26:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-16T11:26:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_34': 2023-12-16T11:26:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:26:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:26:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:26:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 13f7849dbd4b 2023-12-16T11:26:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.13f7849dbd4b 2023-12-16T11:26:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:26:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:26:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:26:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:26:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:17 2023 2023-12-16T11:26:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:17 2023 2023-12-16T11:26:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:17 2023 2023-12-16T11:26:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:26:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:26:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 2c0234b3-7175-40ed-8696-d104af2ef26b 2023-12-16T11:26:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:26:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:26:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-16T11:26:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:26:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:26:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=10 2023-12-16T11:26:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:26:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:26:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 44728320 bytes 2023-12-16T11:26:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 49158144 bytes 2023-12-16T11:26:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 93373440 bytes 2023-12-16T11:26:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 175724544 bytes 2023-12-16T11:26:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 188939264 bytes 2023-12-16T11:26:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 271935488 bytes 2023-12-16T11:26:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 314436608 bytes 2023-12-16T11:26:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 348851200 bytes 2023-12-16T11:26:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 394256384 bytes 2023-12-16T11:26:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 414353408 bytes 2023-12-16T11:26:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 479250432 bytes 2023-12-16T11:26:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 527551488 bytes 2023-12-16T11:26:20.528 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5994178713274698) > weight (0.25) 2023-12-16T11:26:20.529 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5323718293345712) > weight (0.25) 2023-12-16T11:26:20.529 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9156885368394134) > weight (0.25) 2023-12-16T11:26:20.529 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-16T11:26:20.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-16T11:26:20.529 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-16T11:26:20.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:26:20.526+0000 7efce7eaa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 91736) UID: 1000 2023-12-16T11:26:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 629744640 bytes 2023-12-16T11:26:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 703084544 bytes 2023-12-16T11:26:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 735462400 bytes 2023-12-16T11:26:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 799772672 bytes 2023-12-16T11:26:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 825779200 bytes 2023-12-16T11:26:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 828754944 bytes 2023-12-16T11:26:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 866190336 bytes 2023-12-16T11:26:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 925502464 bytes 2023-12-16T11:26:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 963081216 bytes 2023-12-16T11:26:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1037503488 bytes 2023-12-16T11:26:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1063446528 bytes 2023-12-16T11:26:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1082312704 bytes 2023-12-16T11:26:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1106710528 bytes 2023-12-16T11:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1118108672 bytes 2023-12-16T11:26:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1125832704 bytes 2023-12-16T11:26:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1148681216 bytes 2023-12-16T11:26:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1223577600 bytes 2023-12-16T11:26:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1248094208 bytes 2023-12-16T11:26:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1269334016 bytes 2023-12-16T11:26:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1317227520 bytes 2023-12-16T11:26:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1358263296 bytes 2023-12-16T11:26:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1384331264 bytes 2023-12-16T11:26:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1395653632 bytes 2023-12-16T11:26:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1422252032 bytes 2023-12-16T11:26:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1489935360 bytes 2023-12-16T11:26:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1535546368 bytes 2023-12-16T11:26:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1605326848 bytes 2023-12-16T11:26:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1605920768 bytes 2023-12-16T11:26:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1666132992 bytes 2023-12-16T11:26:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1673609216 bytes 2023-12-16T11:26:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1750270976 bytes 2023-12-16T11:26:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1828384768 bytes 2023-12-16T11:26:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1843348480 bytes 2023-12-16T11:26:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1885385728 bytes 2023-12-16T11:26:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1935627264 bytes 2023-12-16T11:26:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1958224896 bytes 2023-12-16T11:26:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1974833152 bytes 2023-12-16T11:26:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2036878336 bytes 2023-12-16T11:26:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2113760256 bytes 2023-12-16T11:26:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2182269952 bytes 2023-12-16T11:26:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2240260096 bytes 2023-12-16T11:26:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2301086720 bytes 2023-12-16T11:26:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_35 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-16T11:26:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:26:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_35': 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 140294efb7a 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.140294efb7a 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:26:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:28 2023 2023-12-16T11:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:28 2023 2023-12-16T11:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:28 2023 2023-12-16T11:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 9e69de8c-0a99-4a9c-9e97-10fd69295190 2023-12-16T11:26:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:26:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:26:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-16T11:26:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:26:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:26:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:26:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:26:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:26:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 27750400 bytes 2023-12-16T11:26:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 85870592 bytes 2023-12-16T11:26:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 116392960 bytes 2023-12-16T11:26:29.530 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-16T11:26:29.530 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:26:29.530 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-16T11:26:29.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-16T11:26:29.531 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-16T11:26:29.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-16T11:26:29.534 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-16T11:26:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 161797120 bytes 2023-12-16T11:26:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 255492096 bytes 2023-12-16T11:26:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 306718720 bytes 2023-12-16T11:26:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 407491584 bytes 2023-12-16T11:26:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 486147072 bytes 2023-12-16T11:26:30.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 548615168 bytes 2023-12-16T11:26:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 557262848 bytes 2023-12-16T11:26:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 559011840 bytes 2023-12-16T11:26:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 602881024 bytes 2023-12-16T11:26:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 627297280 bytes 2023-12-16T11:26:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 645908480 bytes 2023-12-16T11:26:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 689398784 bytes 2023-12-16T11:26:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 779042816 bytes 2023-12-16T11:26:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 864431104 bytes 2023-12-16T11:26:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 876250112 bytes 2023-12-16T11:26:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 913355776 bytes 2023-12-16T11:26:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 974764032 bytes 2023-12-16T11:26:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 993278976 bytes 2023-12-16T11:26:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1037474816 bytes 2023-12-16T11:26:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1059062784 bytes 2023-12-16T11:26:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1113582592 bytes 2023-12-16T11:26:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1158596608 bytes 2023-12-16T11:26:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1196344320 bytes 2023-12-16T11:26:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1215800320 bytes 2023-12-16T11:26:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1281916928 bytes 2023-12-16T11:26:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1373511680 bytes 2023-12-16T11:26:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1401372672 bytes 2023-12-16T11:26:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1490994176 bytes 2023-12-16T11:26:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1550568448 bytes 2023-12-16T11:26:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1551421440 bytes 2023-12-16T11:26:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1593109504 bytes 2023-12-16T11:26:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1631322112 bytes 2023-12-16T11:26:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1666924544 bytes 2023-12-16T11:26:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1689602048 bytes 2023-12-16T11:26:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1763461120 bytes 2023-12-16T11:26:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1790290944 bytes 2023-12-16T11:26:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1822300160 bytes 2023-12-16T11:26:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1904677888 bytes 2023-12-16T11:26:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1978804224 bytes 2023-12-16T11:26:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 2031391744 bytes 2023-12-16T11:26:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2036839424 bytes 2023-12-16T11:26:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2040352768 bytes 2023-12-16T11:26:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2135314432 bytes 2023-12-16T11:26:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2180052992 bytes 2023-12-16T11:26:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2222222336 bytes 2023-12-16T11:26:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2268769280 bytes 2023-12-16T11:26:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2289069056 bytes 2023-12-16T11:26:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2334786560 bytes 2023-12-16T11:26:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2385565696 bytes 2023-12-16T11:26:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2407705600 bytes 2023-12-16T11:26:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2416680960 bytes 2023-12-16T11:26:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2482801664 bytes 2023-12-16T11:26:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2524144640 bytes 2023-12-16T11:26:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2560764928 bytes 2023-12-16T11:26:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2606606336 bytes 2023-12-16T11:26:39.784 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:26:39.779+0000 7f6f756e7640 -1 reset not still connected to 0x5577e9416a90 2023-12-16T11:26:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2663595008 bytes 2023-12-16T11:26:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2713607168 bytes 2023-12-16T11:26:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2714209280 bytes 2023-12-16T11:26:40.363 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:26:40.359+0000 7f6f756e7640 -1 reset not still connected to 0x5577e9416a90 2023-12-16T11:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2761060352 bytes 2023-12-16T11:26:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2852222976 bytes 2023-12-16T11:26:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_36 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:26:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-16T11:26:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:26:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-16T11:26:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_36': 2023-12-16T11:26:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 14146b30ec69 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.14146b30ec69 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:40 2023 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:40 2023 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:40 2023 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 6b19399b-d04e-436d-88a4-a836895fcf94 2023-12-16T11:26:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:26:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:26:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-16T11:26:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:26:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:26:42.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:26:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:26:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:26:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 9503744 bytes 2023-12-16T11:26:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 35733504 bytes 2023-12-16T11:26:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 144990208 bytes 2023-12-16T11:26:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 189193216 bytes 2023-12-16T11:26:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 231871488 bytes 2023-12-16T11:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 319493120 bytes 2023-12-16T11:26:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 379247616 bytes 2023-12-16T11:26:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 430793728 bytes 2023-12-16T11:26:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 434975744 bytes 2023-12-16T11:26:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 461207552 bytes 2023-12-16T11:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 498014208 bytes 2023-12-16T11:26:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 537640960 bytes 2023-12-16T11:26:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 570699776 bytes 2023-12-16T11:26:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 608050176 bytes 2023-12-16T11:26:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 693115904 bytes 2023-12-16T11:26:45.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 715845632 bytes 2023-12-16T11:26:45.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 801269760 bytes 2023-12-16T11:26:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 836978688 bytes 2023-12-16T11:26:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 837579776 bytes 2023-12-16T11:26:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 860143616 bytes 2023-12-16T11:26:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 912202752 bytes 2023-12-16T11:26:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 912813056 bytes 2023-12-16T11:26:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 987303936 bytes 2023-12-16T11:26:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1004109824 bytes 2023-12-16T11:26:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1021864960 bytes 2023-12-16T11:26:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1062672384 bytes 2023-12-16T11:26:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1080665088 bytes 2023-12-16T11:26:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1147138048 bytes 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_37 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:26:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:26:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-16T11:26:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:26:48.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-16T11:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_37': 2023-12-16T11:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 141d1fd4ab48 2023-12-16T11:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.141d1fd4ab48 2023-12-16T11:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:47 2023 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:47 2023 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:47 2023 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 568c8cb4-1132-4824-acb5-5d73627a32e9 2023-12-16T11:26:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-16T11:26:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:26:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=10 2023-12-16T11:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:26:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:26:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 49226752 bytes 2023-12-16T11:26:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 60573696 bytes 2023-12-16T11:26:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 124370944 bytes 2023-12-16T11:26:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 215055360 bytes 2023-12-16T11:26:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 238441472 bytes 2023-12-16T11:26:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 256585728 bytes 2023-12-16T11:26:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 309381120 bytes 2023-12-16T11:26:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 380718080 bytes 2023-12-16T11:26:49.902 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 458371072 bytes 2023-12-16T11:26:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 496594944 bytes 2023-12-16T11:26:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 528436224 bytes 2023-12-16T11:26:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 562941952 bytes 2023-12-16T11:26:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 602509312 bytes 2023-12-16T11:26:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 626016256 bytes 2023-12-16T11:26:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 661100544 bytes 2023-12-16T11:26:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 679916544 bytes 2023-12-16T11:26:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 769350656 bytes 2023-12-16T11:26:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 793823232 bytes 2023-12-16T11:26:51.541 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-16T11:26:51.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-16T11:26:51.542 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-16T11:26:51.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:26:51.538+0000 7ff08e807640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 100011) UID: 1000 2023-12-16T11:26:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 833790976 bytes 2023-12-16T11:26:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 894258176 bytes 2023-12-16T11:26:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 963963904 bytes 2023-12-16T11:26:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 973083648 bytes 2023-12-16T11:26:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 984229888 bytes 2023-12-16T11:26:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1018417152 bytes 2023-12-16T11:26:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1067543552 bytes 2023-12-16T11:26:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1171277824 bytes 2023-12-16T11:26:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1252465664 bytes 2023-12-16T11:26:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1301578752 bytes 2023-12-16T11:26:53.543 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-16T11:26:53.543 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:26:53.543 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-16T11:26:53.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-16T11:26:53.544 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-16T11:26:53.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-16T11:26:53.548 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-16T11:26:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1347907584 bytes 2023-12-16T11:26:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1374660608 bytes 2023-12-16T11:26:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1426954240 bytes 2023-12-16T11:26:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1451718656 bytes 2023-12-16T11:26:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1508081664 bytes 2023-12-16T11:26:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1520018432 bytes 2023-12-16T11:26:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1580736512 bytes 2023-12-16T11:26:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1601187840 bytes 2023-12-16T11:26:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1613607936 bytes 2023-12-16T11:26:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1646043136 bytes 2023-12-16T11:26:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1716330496 bytes 2023-12-16T11:26:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1774396416 bytes 2023-12-16T11:26:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1785464832 bytes 2023-12-16T11:26:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1860107264 bytes 2023-12-16T11:26:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1906478080 bytes 2023-12-16T11:26:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1937034240 bytes 2023-12-16T11:26:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2025159680 bytes 2023-12-16T11:26:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2032502784 bytes 2023-12-16T11:26:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2105394176 bytes 2023-12-16T11:26:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2135205888 bytes 2023-12-16T11:26:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2187175936 bytes 2023-12-16T11:26:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2316029952 bytes 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_38 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:26:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:26:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-16T11:26:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:26:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-16T11:26:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_38': 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 142cf9ae53f8 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.142cf9ae53f8 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:58 2023 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:58 2023 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:58 2023 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:26:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:26:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 53afa996-62cf-4e71-8df9-ce49cbffdd96 2023-12-16T11:26:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:26:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:26:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-16T11:26:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:26:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:26:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:26:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:26:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:26:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:26:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:26:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 80721920 bytes 2023-12-16T11:26:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 131883008 bytes 2023-12-16T11:26:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 176362496 bytes 2023-12-16T11:26:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 193884160 bytes 2023-12-16T11:26:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 203130880 bytes 2023-12-16T11:26:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 216312832 bytes 2023-12-16T11:27:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 265879552 bytes 2023-12-16T11:27:00.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 299533312 bytes 2023-12-16T11:27:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 376399872 bytes 2023-12-16T11:27:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 402117632 bytes 2023-12-16T11:27:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 503949312 bytes 2023-12-16T11:27:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 558768128 bytes 2023-12-16T11:27:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 615409664 bytes 2023-12-16T11:27:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 650935296 bytes 2023-12-16T11:27:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 696487936 bytes 2023-12-16T11:27:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 736232448 bytes 2023-12-16T11:27:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 776239104 bytes 2023-12-16T11:27:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 808819712 bytes 2023-12-16T11:27:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 827078656 bytes 2023-12-16T11:27:02.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 871193600 bytes 2023-12-16T11:27:02.855 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 939308032 bytes 2023-12-16T11:27:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 948211712 bytes 2023-12-16T11:27:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1007418368 bytes 2023-12-16T11:27:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1055384576 bytes 2023-12-16T11:27:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1070500864 bytes 2023-12-16T11:27:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1123736576 bytes 2023-12-16T11:27:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1178968064 bytes 2023-12-16T11:27:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1217437696 bytes 2023-12-16T11:27:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1218061312 bytes 2023-12-16T11:27:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1261706240 bytes 2023-12-16T11:27:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1318560768 bytes 2023-12-16T11:27:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1335740416 bytes 2023-12-16T11:27:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1342374912 bytes 2023-12-16T11:27:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1355502592 bytes 2023-12-16T11:27:05.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1373443072 bytes 2023-12-16T11:27:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1448973312 bytes 2023-12-16T11:27:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1549346816 bytes 2023-12-16T11:27:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1639822336 bytes 2023-12-16T11:27:06.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1680146432 bytes 2023-12-16T11:27:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1721409536 bytes 2023-12-16T11:27:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1786076160 bytes 2023-12-16T11:27:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1817717760 bytes 2023-12-16T11:27:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1843166208 bytes 2023-12-16T11:27:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1928697856 bytes 2023-12-16T11:27:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1970309120 bytes 2023-12-16T11:27:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2019828736 bytes 2023-12-16T11:27:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2038722560 bytes 2023-12-16T11:27:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2081162240 bytes 2023-12-16T11:27:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2110273536 bytes 2023-12-16T11:27:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2141363200 bytes 2023-12-16T11:27:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2183290880 bytes 2023-12-16T11:27:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2282213376 bytes 2023-12-16T11:27:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2352519168 bytes 2023-12-16T11:27:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2407562240 bytes 2023-12-16T11:27:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2439320576 bytes 2023-12-16T11:27:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2457506816 bytes 2023-12-16T11:27:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2563237888 bytes 2023-12-16T11:27:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2583326720 bytes 2023-12-16T11:27:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2583958528 bytes 2023-12-16T11:27:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2620575744 bytes 2023-12-16T11:27:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2737507328 bytes 2023-12-16T11:27:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2771955712 bytes 2023-12-16T11:27:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2846214144 bytes 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_39 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:27:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2023-12-16T11:27:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:27:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:27:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-16T11:27:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:27:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-16T11:27:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:27:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-16T11:27:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_39': 2023-12-16T11:27:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:27:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:27:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1435a413701f 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1435a413701f 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:11 2023 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:11 2023 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:11 2023 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3c59eac5-6520-4cf7-9b72-d0a11d9b9398 2023-12-16T11:27:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:27:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:27:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-16T11:27:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:27:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:27:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:27:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:27:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:27:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 102567936 bytes 2023-12-16T11:27:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 180823040 bytes 2023-12-16T11:27:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 222167040 bytes 2023-12-16T11:27:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 235800576 bytes 2023-12-16T11:27:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 246773760 bytes 2023-12-16T11:27:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 276027392 bytes 2023-12-16T11:27:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 297716736 bytes 2023-12-16T11:27:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 302434304 bytes 2023-12-16T11:27:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 362987520 bytes 2023-12-16T11:27:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 420980736 bytes 2023-12-16T11:27:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 438548480 bytes 2023-12-16T11:27:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 461940736 bytes 2023-12-16T11:27:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 476410880 bytes 2023-12-16T11:27:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 513715200 bytes 2023-12-16T11:27:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 514342912 bytes 2023-12-16T11:27:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 552455168 bytes 2023-12-16T11:27:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 570609664 bytes 2023-12-16T11:27:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 611990528 bytes 2023-12-16T11:27:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 675328000 bytes 2023-12-16T11:27:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 681511936 bytes 2023-12-16T11:27:14.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 711615488 bytes 2023-12-16T11:27:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 751712256 bytes 2023-12-16T11:27:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 799290368 bytes 2023-12-16T11:27:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 809656320 bytes 2023-12-16T11:27:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 900366336 bytes 2023-12-16T11:27:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 929738752 bytes 2023-12-16T11:27:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 971433984 bytes 2023-12-16T11:27:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 985699328 bytes 2023-12-16T11:27:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 992702464 bytes 2023-12-16T11:27:16.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1031975936 bytes 2023-12-16T11:27:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1096083456 bytes 2023-12-16T11:27:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1133228032 bytes 2023-12-16T11:27:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1175485440 bytes 2023-12-16T11:27:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1212733440 bytes 2023-12-16T11:27:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1302882304 bytes 2023-12-16T11:27:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1358011392 bytes 2023-12-16T11:27:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1395906560 bytes 2023-12-16T11:27:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1428539392 bytes 2023-12-16T11:27:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1491899392 bytes 2023-12-16T11:27:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1510344704 bytes 2023-12-16T11:27:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1558106112 bytes 2023-12-16T11:27:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1624609792 bytes 2023-12-16T11:27:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1686596608 bytes 2023-12-16T11:27:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1711249408 bytes 2023-12-16T11:27:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1727737856 bytes 2023-12-16T11:27:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1732604928 bytes 2023-12-16T11:27:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1829486592 bytes 2023-12-16T11:27:19.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1867510784 bytes 2023-12-16T11:27:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1872881664 bytes 2023-12-16T11:27:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1928171520 bytes 2023-12-16T11:27:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1984998400 bytes 2023-12-16T11:27:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2016418816 bytes 2023-12-16T11:27:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2061122560 bytes 2023-12-16T11:27:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2066469888 bytes 2023-12-16T11:27:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2088907776 bytes 2023-12-16T11:27:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2139909120 bytes 2023-12-16T11:27:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2161725440 bytes 2023-12-16T11:27:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2201719808 bytes 2023-12-16T11:27:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2252142592 bytes 2023-12-16T11:27:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2252736512 bytes 2023-12-16T11:27:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2263790592 bytes 2023-12-16T11:27:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2314966016 bytes 2023-12-16T11:27:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2318099456 bytes 2023-12-16T11:27:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2399901696 bytes 2023-12-16T11:27:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2480830464 bytes 2023-12-16T11:27:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2507083776 bytes 2023-12-16T11:27:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2536161280 bytes 2023-12-16T11:27:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2573698048 bytes 2023-12-16T11:27:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2592600064 bytes 2023-12-16T11:27:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_40 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:27:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-16T11:27:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:27:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-16T11:27:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_40': 2023-12-16T11:27:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 143e30962218 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.143e30962218 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:23 2023 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:23 2023 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:23 2023 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:27:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 05144207-c91b-49f6-a34c-04c0264e6d50 2023-12-16T11:27:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:27:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:27:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-16T11:27:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:27:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:27:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:27:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:27:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:27:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 5789696 bytes 2023-12-16T11:27:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 46058496 bytes 2023-12-16T11:27:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 71558144 bytes 2023-12-16T11:27:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 114539520 bytes 2023-12-16T11:27:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 126558208 bytes 2023-12-16T11:27:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 194588672 bytes 2023-12-16T11:27:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 215472128 bytes 2023-12-16T11:27:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 250054656 bytes 2023-12-16T11:27:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 271030272 bytes 2023-12-16T11:27:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 359250944 bytes 2023-12-16T11:27:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 413572096 bytes 2023-12-16T11:27:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 421870592 bytes 2023-12-16T11:27:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 458583040 bytes 2023-12-16T11:27:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 501767168 bytes 2023-12-16T11:27:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 567518208 bytes 2023-12-16T11:27:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 642117632 bytes 2023-12-16T11:27:27.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 672263168 bytes 2023-12-16T11:27:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 672861184 bytes 2023-12-16T11:27:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 690571264 bytes 2023-12-16T11:27:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 783732736 bytes 2023-12-16T11:27:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 836528128 bytes 2023-12-16T11:27:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 837207040 bytes 2023-12-16T11:27:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 877045760 bytes 2023-12-16T11:27:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 982480896 bytes 2023-12-16T11:27:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1003125760 bytes 2023-12-16T11:27:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1005785088 bytes 2023-12-16T11:27:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1023982592 bytes 2023-12-16T11:27:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1086690304 bytes 2023-12-16T11:27:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1121609728 bytes 2023-12-16T11:27:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1156853760 bytes 2023-12-16T11:27:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1175277568 bytes 2023-12-16T11:27:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1192200192 bytes 2023-12-16T11:27:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1235167232 bytes 2023-12-16T11:27:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1319130112 bytes 2023-12-16T11:27:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1391982592 bytes 2023-12-16T11:27:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1427985408 bytes 2023-12-16T11:27:30.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1453253632 bytes 2023-12-16T11:27:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1485552640 bytes 2023-12-16T11:27:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1498572800 bytes 2023-12-16T11:27:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1540500480 bytes 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_41 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-16T11:27:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:27:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:27:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:27:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2023-12-16T11:27:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:27:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:27:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-16T11:27:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:27:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-16T11:27:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:27:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_41': 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 14468ac462fe 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.14468ac462fe 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:31 2023 2023-12-16T11:27:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:31 2023 2023-12-16T11:27:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:31 2023 2023-12-16T11:27:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:27:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:27:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 331f3a7a-6d77-4b49-9624-c90c97974b2f 2023-12-16T11:27:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:27:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:27:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-16T11:27:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:27:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:27:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=8 2023-12-16T11:27:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:27:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:27:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 53009408 bytes 2023-12-16T11:27:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 75868160 bytes 2023-12-16T11:27:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 166643712 bytes 2023-12-16T11:27:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 167946240 bytes 2023-12-16T11:27:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 222752768 bytes 2023-12-16T11:27:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 267553792 bytes 2023-12-16T11:27:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 301796352 bytes 2023-12-16T11:27:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 344987648 bytes 2023-12-16T11:27:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 423132160 bytes 2023-12-16T11:27:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 423741440 bytes 2023-12-16T11:27:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 465162240 bytes 2023-12-16T11:27:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 483802112 bytes 2023-12-16T11:27:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 573235200 bytes 2023-12-16T11:27:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 638209024 bytes 2023-12-16T11:27:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 742339584 bytes 2023-12-16T11:27:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 759178240 bytes 2023-12-16T11:27:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 821098496 bytes 2023-12-16T11:27:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 857072640 bytes 2023-12-16T11:27:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 908493824 bytes 2023-12-16T11:27:36.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 926273536 bytes 2023-12-16T11:27:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 979062784 bytes 2023-12-16T11:27:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1049537536 bytes 2023-12-16T11:27:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1149215744 bytes 2023-12-16T11:27:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1160241152 bytes 2023-12-16T11:27:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1181156352 bytes 2023-12-16T11:27:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1219636224 bytes 2023-12-16T11:27:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1267286016 bytes 2023-12-16T11:27:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1314756608 bytes 2023-12-16T11:27:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1357093888 bytes 2023-12-16T11:27:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1400738816 bytes 2023-12-16T11:27:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1454232576 bytes 2023-12-16T11:27:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1523321856 bytes 2023-12-16T11:27:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1564595200 bytes 2023-12-16T11:27:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1608738816 bytes 2023-12-16T11:27:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1696747520 bytes 2023-12-16T11:27:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1736253440 bytes 2023-12-16T11:27:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1772023808 bytes 2023-12-16T11:27:39.656 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1840276480 bytes 2023-12-16T11:27:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1890661376 bytes 2023-12-16T11:27:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1904249856 bytes 2023-12-16T11:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_42 2023-12-16T11:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2023-12-16T11:27:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:27:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:27:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-16T11:27:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:27:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-16T11:27:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:27:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-16T11:27:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_42': 2023-12-16T11:27:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:27:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:27:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:27:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 144eebbbc691 2023-12-16T11:27:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.144eebbbc691 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:40 2023 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:40 2023 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:40 2023 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:27:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f91fe4e5-2eb7-4af5-b53e-752782a7a471 2023-12-16T11:27:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:27:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:27:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-16T11:27:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:27:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:27:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:27:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:27:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:27:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 34905088 bytes 2023-12-16T11:27:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 74774528 bytes 2023-12-16T11:27:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 88178688 bytes 2023-12-16T11:27:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 150276096 bytes 2023-12-16T11:27:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 176173056 bytes 2023-12-16T11:27:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 219865088 bytes 2023-12-16T11:27:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 228088832 bytes 2023-12-16T11:27:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 236550144 bytes 2023-12-16T11:27:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 272795648 bytes 2023-12-16T11:27:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 299310080 bytes 2023-12-16T11:27:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 365144064 bytes 2023-12-16T11:27:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 392567808 bytes 2023-12-16T11:27:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 442369024 bytes 2023-12-16T11:27:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 478860288 bytes 2023-12-16T11:27:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 522656768 bytes 2023-12-16T11:27:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 544314368 bytes 2023-12-16T11:27:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 566748160 bytes 2023-12-16T11:27:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 633902080 bytes 2023-12-16T11:27:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 692946944 bytes 2023-12-16T11:27:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 748632064 bytes 2023-12-16T11:27:44.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 783849472 bytes 2023-12-16T11:27:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 787060736 bytes 2023-12-16T11:27:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 827730944 bytes 2023-12-16T11:27:45.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 849393664 bytes 2023-12-16T11:27:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 865156096 bytes 2023-12-16T11:27:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 911963136 bytes 2023-12-16T11:27:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 959513600 bytes 2023-12-16T11:27:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 999701504 bytes 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_43 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:27:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:27:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:27:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2023-12-16T11:27:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:27:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:27:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-16T11:27:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:27:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-16T11:27:46.534 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9139423601885995) > weight (0.25) 2023-12-16T11:27:46.535 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3710903814469175) > weight (0.25) 2023-12-16T11:27:46.535 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.49808468583395193) > weight (0.25) 2023-12-16T11:27:46.535 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9885340884704507) > weight (0.25) 2023-12-16T11:27:46.535 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-12-16T11:27:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:27:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-16T11:27:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_43': 2023-12-16T11:27:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:27:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:27:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1457e3d1ea98 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1457e3d1ea98 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:46 2023 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:46 2023 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:46 2023 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 90080d20-b9fb-4d39-8314-78ffc8c74551 2023-12-16T11:27:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:27:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:27:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-16T11:27:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:27:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:27:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=6 2023-12-16T11:27:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:27:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:27:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 13216768 bytes 2023-12-16T11:27:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 40314880 bytes 2023-12-16T11:27:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 80701440 bytes 2023-12-16T11:27:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 117094400 bytes 2023-12-16T11:27:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 182124544 bytes 2023-12-16T11:27:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 206963712 bytes 2023-12-16T11:27:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 249714688 bytes 2023-12-16T11:27:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 291701760 bytes 2023-12-16T11:27:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 399503360 bytes 2023-12-16T11:27:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 412943360 bytes 2023-12-16T11:27:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 443256832 bytes 2023-12-16T11:27:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 454633472 bytes 2023-12-16T11:27:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 474541056 bytes 2023-12-16T11:27:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 525792256 bytes 2023-12-16T11:27:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 554009600 bytes 2023-12-16T11:27:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 630447104 bytes 2023-12-16T11:27:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 671087616 bytes 2023-12-16T11:27:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 707440640 bytes 2023-12-16T11:27:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 712481792 bytes 2023-12-16T11:27:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 722758656 bytes 2023-12-16T11:27:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 781303808 bytes 2023-12-16T11:27:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 837807104 bytes 2023-12-16T11:27:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 865725440 bytes 2023-12-16T11:27:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 879599616 bytes 2023-12-16T11:27:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 930540544 bytes 2023-12-16T11:27:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 984271872 bytes 2023-12-16T11:27:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1053032448 bytes 2023-12-16T11:27:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1071070208 bytes 2023-12-16T11:27:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1108890624 bytes 2023-12-16T11:27:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1176896512 bytes 2023-12-16T11:27:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1242929152 bytes 2023-12-16T11:27:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1284636672 bytes 2023-12-16T11:27:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1333937152 bytes 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_44 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:27:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2023-12-16T11:27:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:27:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:27:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:27:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-16T11:27:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:27:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:27:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-16T11:27:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:27:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-16T11:27:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_44': 2023-12-16T11:27:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:27:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:27:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:27:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 145f9c0609df 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.145f9c0609df 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:53 2023 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:53 2023 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:53 2023 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 56361eef-6a64-435b-aaca-5b4920602568 2023-12-16T11:27:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:27:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:27:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-16T11:27:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:27:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:27:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:27:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:27:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:27:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 17405952 bytes 2023-12-16T11:27:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 59346944 bytes 2023-12-16T11:27:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 116517888 bytes 2023-12-16T11:27:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 151420928 bytes 2023-12-16T11:27:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 177773568 bytes 2023-12-16T11:27:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 211126272 bytes 2023-12-16T11:27:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 259038208 bytes 2023-12-16T11:27:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 300716032 bytes 2023-12-16T11:27:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 365293568 bytes 2023-12-16T11:27:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 413138944 bytes 2023-12-16T11:27:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 435179520 bytes 2023-12-16T11:27:56.200 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:27:56.195+0000 7ff3e93a3640 -1 reset not still connected to 0x564ab4d03930 2023-12-16T11:27:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 474676224 bytes 2023-12-16T11:27:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 505337856 bytes 2023-12-16T11:27:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 592976896 bytes 2023-12-16T11:27:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 625415168 bytes 2023-12-16T11:27:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 663616512 bytes 2023-12-16T11:27:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 673155072 bytes 2023-12-16T11:27:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 709960704 bytes 2023-12-16T11:27:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 764587008 bytes 2023-12-16T11:27:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 805312512 bytes 2023-12-16T11:27:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 814846976 bytes 2023-12-16T11:27:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 842255360 bytes 2023-12-16T11:27:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 863834112 bytes 2023-12-16T11:27:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 922918912 bytes 2023-12-16T11:27:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 932969472 bytes 2023-12-16T11:27:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 985236480 bytes 2023-12-16T11:27:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 985858048 bytes 2023-12-16T11:27:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1035398144 bytes 2023-12-16T11:27:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1085123584 bytes 2023-12-16T11:27:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1088055296 bytes 2023-12-16T11:27:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1129997312 bytes 2023-12-16T11:27:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1167588352 bytes 2023-12-16T11:27:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1203774464 bytes 2023-12-16T11:28:00.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1272479744 bytes 2023-12-16T11:28:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1344385024 bytes 2023-12-16T11:28:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1365922816 bytes 2023-12-16T11:28:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1410373632 bytes 2023-12-16T11:28:00.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1452908544 bytes 2023-12-16T11:28:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1502862336 bytes 2023-12-16T11:28:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1631705088 bytes 2023-12-16T11:28:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1656968192 bytes 2023-12-16T11:28:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1694695424 bytes 2023-12-16T11:28:01.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1748468736 bytes 2023-12-16T11:28:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1772104704 bytes 2023-12-16T11:28:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1823400960 bytes 2023-12-16T11:28:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1832522752 bytes 2023-12-16T11:28:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1912693760 bytes 2023-12-16T11:28:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1913313280 bytes 2023-12-16T11:28:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1913939968 bytes 2023-12-16T11:28:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1949556736 bytes 2023-12-16T11:28:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2029305856 bytes 2023-12-16T11:28:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2086557696 bytes 2023-12-16T11:28:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2159566848 bytes 2023-12-16T11:28:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2190663680 bytes 2023-12-16T11:28:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2232981504 bytes 2023-12-16T11:28:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2306467840 bytes 2023-12-16T11:28:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2364787712 bytes 2023-12-16T11:28:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2395923456 bytes 2023-12-16T11:28:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2417372160 bytes 2023-12-16T11:28:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2440109056 bytes 2023-12-16T11:28:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2447229952 bytes 2023-12-16T11:28:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2511267840 bytes 2023-12-16T11:28:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2555314176 bytes 2023-12-16T11:28:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2563239936 bytes 2023-12-16T11:28:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2600161280 bytes 2023-12-16T11:28:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2722798592 bytes 2023-12-16T11:28:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_45 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:28:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:28:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:28:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-16T11:28:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:28:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_45': 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 146754cc1295 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.146754cc1295 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:06 2023 2023-12-16T11:28:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:06 2023 2023-12-16T11:28:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:06 2023 2023-12-16T11:28:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:28:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:28:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3aef10c1-3b15-46e5-bf7b-1633092d93ea 2023-12-16T11:28:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:28:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:28:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-16T11:28:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:28:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:28:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=9 2023-12-16T11:28:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:28:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:28:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 42848256 bytes 2023-12-16T11:28:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 70046720 bytes 2023-12-16T11:28:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 119416832 bytes 2023-12-16T11:28:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 146791424 bytes 2023-12-16T11:28:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 197997568 bytes 2023-12-16T11:28:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 236402688 bytes 2023-12-16T11:28:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 352123904 bytes 2023-12-16T11:28:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 389373952 bytes 2023-12-16T11:28:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 468782080 bytes 2023-12-16T11:28:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 510274560 bytes 2023-12-16T11:28:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 548189184 bytes 2023-12-16T11:28:09.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 608078848 bytes 2023-12-16T11:28:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 632865792 bytes 2023-12-16T11:28:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 662753280 bytes 2023-12-16T11:28:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 716141568 bytes 2023-12-16T11:28:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 760666112 bytes 2023-12-16T11:28:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 812435456 bytes 2023-12-16T11:28:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 819021824 bytes 2023-12-16T11:28:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 862809088 bytes 2023-12-16T11:28:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 904000512 bytes 2023-12-16T11:28:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 989777920 bytes 2023-12-16T11:28:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1018165248 bytes 2023-12-16T11:28:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1031915520 bytes 2023-12-16T11:28:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1075695616 bytes 2023-12-16T11:28:11.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1121751040 bytes 2023-12-16T11:28:11.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1153857536 bytes 2023-12-16T11:28:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1180243968 bytes 2023-12-16T11:28:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1212689408 bytes 2023-12-16T11:28:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1234872320 bytes 2023-12-16T11:28:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1267033088 bytes 2023-12-16T11:28:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1292471296 bytes 2023-12-16T11:28:12.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1350782976 bytes 2023-12-16T11:28:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1389205504 bytes 2023-12-16T11:28:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1409311744 bytes 2023-12-16T11:28:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1478131712 bytes 2023-12-16T11:28:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1515458560 bytes 2023-12-16T11:28:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1563301888 bytes 2023-12-16T11:28:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1584747520 bytes 2023-12-16T11:28:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1626508288 bytes 2023-12-16T11:28:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1659379712 bytes 2023-12-16T11:28:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1730462720 bytes 2023-12-16T11:28:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1786311680 bytes 2023-12-16T11:28:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1833989120 bytes 2023-12-16T11:28:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1834608640 bytes 2023-12-16T11:28:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1852042240 bytes 2023-12-16T11:28:15.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1910432768 bytes 2023-12-16T11:28:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1949306880 bytes 2023-12-16T11:28:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1973005312 bytes 2023-12-16T11:28:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2035519488 bytes 2023-12-16T11:28:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_46 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2023-12-16T11:28:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:28:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:28:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-16T11:28:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:28:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-16T11:28:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:28:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_46': 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 146f84504613 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.146f84504613 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:28:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:16 2023 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:16 2023 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:16 2023 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: fd0fbe2d-ac26-40f9-b708-d9544fad07a5 2023-12-16T11:28:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:28:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:28:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-16T11:28:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:28:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:28:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=6 2023-12-16T11:28:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:28:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:28:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 29083648 bytes 2023-12-16T11:28:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 106743808 bytes 2023-12-16T11:28:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 161396736 bytes 2023-12-16T11:28:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 211847168 bytes 2023-12-16T11:28:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 278364160 bytes 2023-12-16T11:28:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 331018240 bytes 2023-12-16T11:28:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 350804992 bytes 2023-12-16T11:28:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 398814208 bytes 2023-12-16T11:28:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 445699072 bytes 2023-12-16T11:28:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 480188416 bytes 2023-12-16T11:28:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 522772480 bytes 2023-12-16T11:28:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 567309312 bytes 2023-12-16T11:28:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 649210880 bytes 2023-12-16T11:28:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 695987200 bytes 2023-12-16T11:28:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 761976832 bytes 2023-12-16T11:28:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 829676544 bytes 2023-12-16T11:28:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 883356672 bytes 2023-12-16T11:28:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 936281088 bytes 2023-12-16T11:28:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 994790400 bytes 2023-12-16T11:28:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1036919808 bytes 2023-12-16T11:28:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1040049152 bytes 2023-12-16T11:28:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1055386624 bytes 2023-12-16T11:28:21.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9303626718253182) > weight (0.25) 2023-12-16T11:28:21.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5890619295403187) > weight (0.25) 2023-12-16T11:28:21.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.460792183972345) > weight (0.25) 2023-12-16T11:28:21.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2523320393079753) > weight (0.25) 2023-12-16T11:28:21.550 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-16T11:28:21.568 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:28:21.563+0000 7f6f756e7640 -1 reset not still connected to 0x5577d85c3520 2023-12-16T11:28:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1103839232 bytes 2023-12-16T11:28:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1118596096 bytes 2023-12-16T11:28:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1149270016 bytes 2023-12-16T11:28:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1274117120 bytes 2023-12-16T11:28:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1309754368 bytes 2023-12-16T11:28:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1353775104 bytes 2023-12-16T11:28:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1388363776 bytes 2023-12-16T11:28:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1401807872 bytes 2023-12-16T11:28:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1457936384 bytes 2023-12-16T11:28:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:28:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:28:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:28:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:28:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_47 2023-12-16T11:28:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:28:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:28:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:28:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:28:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-16T11:28:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:28:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_47': 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1479be72a761 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1479be72a761 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:23 2023 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:23 2023 2023-12-16T11:28:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:23 2023 2023-12-16T11:28:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:28:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:28:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: bca986c6-16b7-4831-979e-917d8898129e 2023-12-16T11:28:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:28:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:28:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-16T11:28:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:28:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:28:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:28:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:28:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:28:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 19231744 bytes 2023-12-16T11:28:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 50351104 bytes 2023-12-16T11:28:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 61156352 bytes 2023-12-16T11:28:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 120164352 bytes 2023-12-16T11:28:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 209607680 bytes 2023-12-16T11:28:25.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 256874496 bytes 2023-12-16T11:28:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 269874176 bytes 2023-12-16T11:28:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 315578368 bytes 2023-12-16T11:28:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 378048512 bytes 2023-12-16T11:28:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 454537216 bytes 2023-12-16T11:28:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 504930304 bytes 2023-12-16T11:28:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 506825728 bytes 2023-12-16T11:28:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 517213184 bytes 2023-12-16T11:28:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 564013056 bytes 2023-12-16T11:28:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 653799424 bytes 2023-12-16T11:28:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 723833856 bytes 2023-12-16T11:28:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 790961152 bytes 2023-12-16T11:28:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 845048832 bytes 2023-12-16T11:28:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 883173376 bytes 2023-12-16T11:28:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 930861056 bytes 2023-12-16T11:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 948283392 bytes 2023-12-16T11:28:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1008665600 bytes 2023-12-16T11:28:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1043391488 bytes 2023-12-16T11:28:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1068752896 bytes 2023-12-16T11:28:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1102041088 bytes 2023-12-16T11:28:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1156262912 bytes 2023-12-16T11:28:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1174701056 bytes 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_48 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:28:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:28:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:28:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2023-12-16T11:28:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:28:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:28:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-16T11:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-16T11:28:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:28:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-16T11:28:30.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_48': 2023-12-16T11:28:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:28:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:28:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:28:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1483f33b28af 2023-12-16T11:28:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1483f33b28af 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:29 2023 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:29 2023 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:29 2023 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:28:30.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 0133838f-72ed-4a6e-b784-f5eb3197806a 2023-12-16T11:28:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:28:30.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-16T11:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:28:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:28:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=8 2023-12-16T11:28:30.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:28:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:28:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 60369920 bytes 2023-12-16T11:28:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 150604800 bytes 2023-12-16T11:28:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 205597696 bytes 2023-12-16T11:28:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 253547520 bytes 2023-12-16T11:28:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 260659200 bytes 2023-12-16T11:28:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 327404544 bytes 2023-12-16T11:28:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 343047168 bytes 2023-12-16T11:28:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 392776704 bytes 2023-12-16T11:28:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 429197312 bytes 2023-12-16T11:28:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 460621824 bytes 2023-12-16T11:28:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 501134336 bytes 2023-12-16T11:28:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 519704576 bytes 2023-12-16T11:28:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 562883584 bytes 2023-12-16T11:28:32.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 595675136 bytes 2023-12-16T11:28:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 648188928 bytes 2023-12-16T11:28:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 679502848 bytes 2023-12-16T11:28:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 790171648 bytes 2023-12-16T11:28:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 836622336 bytes 2023-12-16T11:28:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 850135040 bytes 2023-12-16T11:28:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 861975552 bytes 2023-12-16T11:28:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 906463232 bytes 2023-12-16T11:28:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 975157248 bytes 2023-12-16T11:28:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1028793344 bytes 2023-12-16T11:28:34.974 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1148390400 bytes 2023-12-16T11:28:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1171465216 bytes 2023-12-16T11:28:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1208448000 bytes 2023-12-16T11:28:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1244541952 bytes 2023-12-16T11:28:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1303767040 bytes 2023-12-16T11:28:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1304424448 bytes 2023-12-16T11:28:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1412722688 bytes 2023-12-16T11:28:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1445350400 bytes 2023-12-16T11:28:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1451151360 bytes 2023-12-16T11:28:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1499484160 bytes 2023-12-16T11:28:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1500100608 bytes 2023-12-16T11:28:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1521297408 bytes 2023-12-16T11:28:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1574767616 bytes 2023-12-16T11:28:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1595993088 bytes 2023-12-16T11:28:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1600323584 bytes 2023-12-16T11:28:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1739901952 bytes 2023-12-16T11:28:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1833322496 bytes 2023-12-16T11:28:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1858020352 bytes 2023-12-16T11:28:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:28:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:28:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:28:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_49 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-16T11:28:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-16T11:28:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:28:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_49': 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 148c264206ec 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.148c264206ec 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:28:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:38 2023 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:38 2023 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:38 2023 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: fc94afcb-ef75-4403-83fc-5aacc271b450 2023-12-16T11:28:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:28:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:28:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-16T11:28:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:28:39.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:28:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=13 2023-12-16T11:28:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:28:39.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:28:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 48034816 bytes 2023-12-16T11:28:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 66184192 bytes 2023-12-16T11:28:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 125223936 bytes 2023-12-16T11:28:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 197575680 bytes 2023-12-16T11:28:40.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 248648704 bytes 2023-12-16T11:28:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 277140480 bytes 2023-12-16T11:28:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 292626432 bytes 2023-12-16T11:28:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 317353984 bytes 2023-12-16T11:28:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 332203008 bytes 2023-12-16T11:28:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 371851264 bytes 2023-12-16T11:28:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 419685376 bytes 2023-12-16T11:28:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 475062272 bytes 2023-12-16T11:28:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 503601152 bytes 2023-12-16T11:28:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 579952640 bytes 2023-12-16T11:28:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 624430080 bytes 2023-12-16T11:28:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 659607552 bytes 2023-12-16T11:28:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 705862656 bytes 2023-12-16T11:28:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 710779904 bytes 2023-12-16T11:28:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 736564224 bytes 2023-12-16T11:28:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 773218304 bytes 2023-12-16T11:28:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 822692864 bytes 2023-12-16T11:28:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 867534848 bytes 2023-12-16T11:28:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 876774400 bytes 2023-12-16T11:28:43.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 946505728 bytes 2023-12-16T11:28:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 975096832 bytes 2023-12-16T11:28:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 995945472 bytes 2023-12-16T11:28:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1062410240 bytes 2023-12-16T11:28:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1117934592 bytes 2023-12-16T11:28:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1187287040 bytes 2023-12-16T11:28:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1249986560 bytes 2023-12-16T11:28:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1319573504 bytes 2023-12-16T11:28:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1354168320 bytes 2023-12-16T11:28:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1368128512 bytes 2023-12-16T11:28:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1446830080 bytes 2023-12-16T11:28:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1453217792 bytes 2023-12-16T11:28:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1502227456 bytes 2023-12-16T11:28:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1548436480 bytes 2023-12-16T11:28:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1590888448 bytes 2023-12-16T11:28:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1617180672 bytes 2023-12-16T11:28:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1646055424 bytes 2023-12-16T11:28:46.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1692972032 bytes 2023-12-16T11:28:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1717949440 bytes 2023-12-16T11:28:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1738532864 bytes 2023-12-16T11:28:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1822276608 bytes 2023-12-16T11:28:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1893926912 bytes 2023-12-16T11:28:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1934898176 bytes 2023-12-16T11:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1936527360 bytes 2023-12-16T11:28:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1981883392 bytes 2023-12-16T11:28:47.806 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:28:47.801+0000 7ff3e93a3640 -1 reset not still connected to 0x564ad1b74750 2023-12-16T11:28:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1997606912 bytes 2023-12-16T11:28:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2013050880 bytes 2023-12-16T11:28:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2043421696 bytes 2023-12-16T11:28:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2120323072 bytes 2023-12-16T11:28:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2191544320 bytes 2023-12-16T11:28:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2232470528 bytes 2023-12-16T11:28:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2280709120 bytes 2023-12-16T11:28:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2303431680 bytes 2023-12-16T11:28:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2323723264 bytes 2023-12-16T11:28:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2387493888 bytes 2023-12-16T11:28:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2412685312 bytes 2023-12-16T11:28:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2477092864 bytes 2023-12-16T11:28:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2495937536 bytes 2023-12-16T11:28:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2531143680 bytes 2023-12-16T11:28:50.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2572622848 bytes 2023-12-16T11:28:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2584198144 bytes 2023-12-16T11:28:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2605276160 bytes 2023-12-16T11:28:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2649454592 bytes 2023-12-16T11:28:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2665873408 bytes 2023-12-16T11:28:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2704755712 bytes 2023-12-16T11:28:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2777157632 bytes 2023-12-16T11:28:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2822597632 bytes 2023-12-16T11:28:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 2853835776 bytes 2023-12-16T11:28:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 2864136192 bytes 2023-12-16T11:28:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:28:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:28:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_50 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n snapshot ']' 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 128M ']' 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=128M 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:28:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2023-12-16T11:28:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ is_pool_mirror_mode_image mirror 2023-12-16T11:28:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' image = image ']' 2023-12-16T11:28:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:28:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-16T11:28:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:28:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:28:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-16T11:28:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:28:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-16T11:28:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_50': 2023-12-16T11:28:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:28:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:28:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:28:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 14948e69b4f4 2023-12-16T11:28:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.14948e69b4f4 2023-12-16T11:28:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:52 2023 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:52 2023 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:52 2023 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 0e617b2e-921c-479c-a203-4d86e513e03e 2023-12-16T11:28:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:28:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n '' ']' 2023-12-16T11:28:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-16T11:28:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:28:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:28:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:28:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:28:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:28:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:28:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:28:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 12922880 bytes 2023-12-16T11:28:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 98737152 bytes 2023-12-16T11:28:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 146565120 bytes 2023-12-16T11:28:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 154678272 bytes 2023-12-16T11:28:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 197325824 bytes 2023-12-16T11:28:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 219469824 bytes 2023-12-16T11:28:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 244893696 bytes 2023-12-16T11:28:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 264075264 bytes 2023-12-16T11:28:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 315288576 bytes 2023-12-16T11:28:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 369260544 bytes 2023-12-16T11:28:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 425249792 bytes 2023-12-16T11:28:55.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 466505728 bytes 2023-12-16T11:28:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 495170560 bytes 2023-12-16T11:28:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 527051776 bytes 2023-12-16T11:28:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 560539648 bytes 2023-12-16T11:28:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 569364480 bytes 2023-12-16T11:28:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 644705280 bytes 2023-12-16T11:28:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 680652800 bytes 2023-12-16T11:28:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 701119488 bytes 2023-12-16T11:28:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 746277888 bytes 2023-12-16T11:28:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 803935232 bytes 2023-12-16T11:28:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 823871488 bytes 2023-12-16T11:28:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 855366656 bytes 2023-12-16T11:28:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 920939520 bytes 2023-12-16T11:28:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 997413888 bytes 2023-12-16T11:28:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1030052864 bytes 2023-12-16T11:28:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1115833344 bytes 2023-12-16T11:28:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1167631360 bytes 2023-12-16T11:28:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1218733056 bytes 2023-12-16T11:28:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1265846272 bytes 2023-12-16T11:28:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1307309056 bytes 2023-12-16T11:28:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1347097600 bytes 2023-12-16T11:28:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1362775040 bytes 2023-12-16T11:28:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1400431616 bytes 2023-12-16T11:28:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1462313984 bytes 2023-12-16T11:28:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1518869504 bytes 2023-12-16T11:29:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1535954944 bytes 2023-12-16T11:29:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1563936768 bytes 2023-12-16T11:29:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1623892992 bytes 2023-12-16T11:29:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1643618304 bytes 2023-12-16T11:29:00.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1707133952 bytes 2023-12-16T11:29:00.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1722940416 bytes 2023-12-16T11:29:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1737922560 bytes 2023-12-16T11:29:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1801226240 bytes 2023-12-16T11:29:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1816275968 bytes 2023-12-16T11:29:01.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1876209664 bytes 2023-12-16T11:29:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1924769792 bytes 2023-12-16T11:29:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1929826304 bytes 2023-12-16T11:29:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1970417664 bytes 2023-12-16T11:29:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2016952320 bytes 2023-12-16T11:29:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2017568768 bytes 2023-12-16T11:29:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2046241792 bytes 2023-12-16T11:29:02.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2082036736 bytes 2023-12-16T11:29:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2082662400 bytes 2023-12-16T11:29:03.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2097360896 bytes 2023-12-16T11:29:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2131889152 bytes 2023-12-16T11:29:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2174682112 bytes 2023-12-16T11:29:03.536 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9464766091656223) > weight (0.25) 2023-12-16T11:29:03.536 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8568962172342487) > weight (0.25) 2023-12-16T11:29:03.536 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-16T11:29:03.536 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-16T11:29:03.537 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-16T11:29:03.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:29:03.533+0000 7f7132b42640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 98206) UID: 1000 2023-12-16T11:29:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2206995456 bytes 2023-12-16T11:29:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2266190848 bytes 2023-12-16T11:29:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2300094464 bytes 2023-12-16T11:29:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2326491136 bytes 2023-12-16T11:29:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2333306880 bytes 2023-12-16T11:29:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2361753600 bytes 2023-12-16T11:29:04.537 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-16T11:29:04.537 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:29:04.538 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-16T11:29:04.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-16T11:29:04.538 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-16T11:29:04.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-16T11:29:04.541 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2023-12-16T11:29:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2388718592 bytes 2023-12-16T11:29:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2479835136 bytes 2023-12-16T11:29:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2530897920 bytes 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_count=50 2023-12-16T11:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local count 2023-12-16T11:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_count=0 2023-12-16T11:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 40 2023-12-16T11:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in `seq 1 40` 2023-12-16T11:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 1 -ne 1 2023-12-16T11:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-16T11:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep 'images: ' 2023-12-16T11:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cut '-d ' -f 2 2023-12-16T11:29:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ count=50 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 50 = 50 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state 2023-12-16T11:29:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 40 2023-12-16T11:29:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in `seq 1 40` 2023-12-16T11:29:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 1 -ne 1 2023-12-16T11:29:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-16T11:29:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep 'image health:' 2023-12-16T11:29:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cut '-d ' -f 3 2023-12-16T11:29:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=OK 2023-12-16T11:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test OK = ERROR 2023-12-16T11:29:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test OK = OK 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_count=50 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local count 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_count=0 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 40 2023-12-16T11:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in `seq 1 40` 2023-12-16T11:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 1 -ne 1 2023-12-16T11:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-16T11:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep 'images: ' 2023-12-16T11:29:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cut '-d ' -f 2 2023-12-16T11:29:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ count=50 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 50 = 50 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 40 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in `seq 1 40` 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 1 -ne 1 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-16T11:29:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep 'image health:' 2023-12-16T11:29:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cut '-d ' -f 3 2023-12-16T11:29:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:29:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=WARNING 2023-12-16T11:29:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test WARNING = ERROR 2023-12-16T11:29:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test WARNING = OK 2023-12-16T11:29:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in `seq 1 40` 2023-12-16T11:29:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 2 -ne 1 2023-12-16T11:29:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 30 2023-12-16T11:29:30.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6191584691213484) > weight (0.25) 2023-12-16T11:29:30.551 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-16T11:29:30.551 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-16T11:29:30.551 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-16T11:29:30.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:29:30.548+0000 7f8ce764a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 102132) UID: 1000 2023-12-16T11:29:32.552 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-16T11:29:32.553 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:29:32.553 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-16T11:29:32.553 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:29:32.553 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:29:32.556 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:29:32.556 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-12-16T11:29:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-16T11:29:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep 'image health:' 2023-12-16T11:29:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cut '-d ' -f 3 2023-12-16T11:29:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:29:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=OK 2023-12-16T11:29:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test OK = ERROR 2023-12-16T11:29:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test OK = OK 2023-12-16T11:29:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ testlog 'TEST: compare many images' 2023-12-16T11:29:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee -a /tmp/tmp.jvOAE3Umds/rbd-mirror.test.log 2023-12-16T11:29:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ date '+%F %T' 2023-12-16T11:29:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo 2023-12-16 11:29:35 TEST: compare many images 2023-12-16T11:29:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:2023-12-16 11:29:35 TEST: compare many images 2023-12-16T11:29:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 50 2023-12-16T11:29:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:29:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_1 2023-12-16T11:29:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-16T11:29:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:29:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-16T11:29:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-16T11:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:29:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-16T11:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:29:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:29:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:29:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:29:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:29:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:29:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:29:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-16T11:29:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:29:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:29:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:29:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:29:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:29:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:29:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:29:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-16T11:29:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_1.status 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_1.status 2023-12-16T11:29:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:29:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.status 2023-12-16T11:29:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_1.status 2023-12-16T11:29:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_1.status 2023-12-16T11:29:37.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:29:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.status 2023-12-16T11:29:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-16T11:29:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-16T11:29:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 119 2023-12-16T11:29:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.status 2023-12-16T11:29:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.status 2023-12-16T11:29:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-16T11:29:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:29:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=119 2023-12-16T11:29:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:29:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.status 2023-12-16T11:29:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.status 2023-12-16T11:29:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-16T11:29:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=68 2023-12-16T11:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 119 = 68 2023-12-16T11:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:29:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.status 2023-12-16T11:29:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.status 2023-12-16T11:29:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-16T11:29:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:29:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:29:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:29:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:29:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 119 = '' 2023-12-16T11:29:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:29:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.status 2023-12-16T11:29:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.status 2023-12-16T11:29:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-16T11:29:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=119 2023-12-16T11:29:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 119 = 119 2023-12-16T11:29:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-16T11:29:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:29:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:29:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_1': 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c516d20a8e 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c516d20a8e 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:20:50 2023 2023-12-16T11:29:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:20:50 2023 2023-12-16T11:29:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:20:50 2023 2023-12-16T11:29:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:29:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:29:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:29:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3fc80ffb-aa2e-47fa-aeab-e20034fa097b 2023-12-16T11:29:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:29:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-16T11:29:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.export 2023-12-16T11:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.export 2023-12-16T11:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.export 2023-12-16T11:29:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.export 2023-12-16T11:29:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:29:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.export 2023-12-16T11:29:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:29:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.export 2023-12-16T11:29:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_1.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_1.export 2023-12-16T11:29:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_2 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:29:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-16T11:29:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:29:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-16T11:29:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:29:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:29:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:29:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-16T11:29:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:29:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:29:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:29:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:29:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:29:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:29:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:29:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:29:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-16T11:29:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:29:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:29:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:29:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:29:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:29:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:43.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-16T11:29:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 1 closest matches: 2023-12-16T11:29:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_23 2023-12-16T11:29:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-16T11:29:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:29:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:29:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:29:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-16T11:29:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:29:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:29:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:29:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:29:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_2.status 2023-12-16T11:29:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_2.status 2023-12-16T11:29:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:29:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.status 2023-12-16T11:29:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_2.status 2023-12-16T11:29:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_2.status 2023-12-16T11:29:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:29:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-16T11:29:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-16T11:29:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 121 2023-12-16T11:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.status 2023-12-16T11:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.status 2023-12-16T11:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-16T11:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:29:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=121 2023-12-16T11:29:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:29:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-16T11:29:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=69 2023-12-16T11:29:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 121 = 69 2023-12-16T11:29:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:29:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-16T11:29:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:29:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:29:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:29:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:29:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 121 = '' 2023-12-16T11:29:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:29:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-16T11:29:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:45.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 121 = '' 2023-12-16T11:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:29:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.status 2023-12-16T11:29:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-16T11:29:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=121 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 121 = 121 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:29:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:29:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-16T11:29:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_2': 2023-12-16T11:29:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:29:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:29:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:29:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12095c07ab8 2023-12-16T11:29:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12095c07ab8 2023-12-16T11:29:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:29:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:29:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:29:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:29:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:12 2023 2023-12-16T11:29:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:12 2023 2023-12-16T11:29:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:12 2023 2023-12-16T11:29:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:29:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:29:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:29:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 5c88cdc9-a15c-46ab-8224-c40a8d9af053 2023-12-16T11:29:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-16T11:29:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:29:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:29:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.export 2023-12-16T11:29:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.export 2023-12-16T11:29:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.export 2023-12-16T11:29:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.export 2023-12-16T11:29:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:29:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.export 2023-12-16T11:29:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:29:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.export 2023-12-16T11:29:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_2.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_2.export 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_3 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:29:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-16T11:29:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:29:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-16T11:29:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:29:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:29:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:29:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:29:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:29:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:29:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-16T11:29:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:29:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:29:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:29:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:29:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:29:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:29:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:29:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_3.status 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_3.status 2023-12-16T11:29:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:29:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.status 2023-12-16T11:29:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_3.status 2023-12-16T11:29:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_3.status 2023-12-16T11:29:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.status 2023-12-16T11:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-16T11:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-16T11:29:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 123 2023-12-16T11:29:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.status 2023-12-16T11:29:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.status 2023-12-16T11:29:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-16T11:29:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:29:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=123 2023-12-16T11:29:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:29:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.status 2023-12-16T11:29:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.status 2023-12-16T11:29:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-16T11:29:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:51.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:29:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:29:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:29:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:29:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 123 = '' 2023-12-16T11:29:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:29:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.status 2023-12-16T11:29:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.status 2023-12-16T11:29:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-16T11:29:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:29:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:29:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:29:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:29:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:29:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 123 = '' 2023-12-16T11:29:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:29:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:29:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.status 2023-12-16T11:29:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.status 2023-12-16T11:29:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-16T11:29:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:29:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=123 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 123 = 123 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:29:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:29:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_3': 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c0f0c21fc5 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c0f0c21fc5 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:29:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:11 2023 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:11 2023 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:11 2023 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3e35d21e-975d-43a4-a669-8eaeaf673ca7 2023-12-16T11:29:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.export 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.export 2023-12-16T11:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.export 2023-12-16T11:29:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.export 2023-12-16T11:29:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:29:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.export 2023-12-16T11:29:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:29:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.export 2023-12-16T11:29:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_3.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_3.export 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_4 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:29:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:29:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-16T11:29:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:29:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-16T11:29:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:29:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:29:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:29:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:29:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-16T11:29:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:29:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:29:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:29:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:29:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:29:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:29:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:29:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:29:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:29:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-16T11:29:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:29:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:29:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:29:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:29:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-16T11:29:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 3 closest matches: 2023-12-16T11:29:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_40 2023-12-16T11:29:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_44 2023-12-16T11:29:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_48 2023-12-16T11:29:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-16T11:29:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:29:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:29:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:29:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:29:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-16T11:29:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-16T11:29:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:29:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:29:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:29:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:29:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:29:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:29:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:29:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-16T11:29:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 2 closest matches: 2023-12-16T11:29:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_42 2023-12-16T11:29:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_46 2023-12-16T11:29:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:29:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:29:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:29:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:29:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:29:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:29:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:29:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:29:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-16T11:29:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 3 closest matches: 2023-12-16T11:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_41 2023-12-16T11:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_45 2023-12-16T11:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_49 2023-12-16T11:29:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 1 2023-12-16T11:29:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:29:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:29:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-16T11:29:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:29:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:29:57.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:29:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-16T11:29:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 3 closest matches: 2023-12-16T11:29:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_40 2023-12-16T11:29:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_44 2023-12-16T11:29:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_48 2023-12-16T11:29:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:29:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:29:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:29:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:29:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:29:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:29:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-16T11:29:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-16T11:29:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:29:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:29:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:29:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:29:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:29:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:29:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:29:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:29:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-16T11:29:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 2 closest matches: 2023-12-16T11:29:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_42 2023-12-16T11:29:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_46 2023-12-16T11:29:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-16T11:29:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:29:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:29:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:29:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:29:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:29:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:29:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:29:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-16T11:29:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 3 closest matches: 2023-12-16T11:29:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_41 2023-12-16T11:29:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_45 2023-12-16T11:29:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_49 2023-12-16T11:29:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:29:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:29:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 2 2023-12-16T11:29:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:29:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:29:59.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:29:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:29:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:29:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:29:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-16T11:30:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 3 closest matches: 2023-12-16T11:30:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_40 2023-12-16T11:30:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_44 2023-12-16T11:30:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_48 2023-12-16T11:30:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-16T11:30:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-16T11:30:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-16T11:30:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:30:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:30:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:30:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-16T11:30:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 2 closest matches: 2023-12-16T11:30:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_42 2023-12-16T11:30:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_46 2023-12-16T11:30:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-16T11:30:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:30:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:30:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:30:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:30:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:30:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-16T11:30:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 3 closest matches: 2023-12-16T11:30:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_41 2023-12-16T11:30:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_45 2023-12-16T11:30:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_49 2023-12-16T11:30:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 4 2023-12-16T11:30:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-16T11:30:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 3 closest matches: 2023-12-16T11:30:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_40 2023-12-16T11:30:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_44 2023-12-16T11:30:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_48 2023-12-16T11:30:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-16T11:30:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:no valid command found; 1 closest matches: 2023-12-16T11:30:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status mirror/image_47 2023-12-16T11:30:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:30:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:30:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:30:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-16T11:30:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_4.status 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_4.status 2023-12-16T11:30:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.status 2023-12-16T11:30:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_4.status 2023-12-16T11:30:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_4.status 2023-12-16T11:30:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.status 2023-12-16T11:30:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-16T11:30:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-16T11:30:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 125 2023-12-16T11:30:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.status 2023-12-16T11:30:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.status 2023-12-16T11:30:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-16T11:30:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=125 2023-12-16T11:30:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.status 2023-12-16T11:30:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.status 2023-12-16T11:30:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-16T11:30:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 125 = '' 2023-12-16T11:30:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.status 2023-12-16T11:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.status 2023-12-16T11:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-16T11:30:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 125 = '' 2023-12-16T11:30:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.status 2023-12-16T11:30:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.status 2023-12-16T11:30:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-16T11:30:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=125 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 125 = 125 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-16T11:30:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_4': 2023-12-16T11:30:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129ce41507ec 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129ce41507ec 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:17 2023 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:17 2023 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:17 2023 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f9aaccb8-5e33-401a-b44f-3e6c030d0568 2023-12-16T11:30:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-16T11:30:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:30:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.export 2023-12-16T11:30:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.export 2023-12-16T11:30:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.export 2023-12-16T11:30:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.export 2023-12-16T11:30:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.export 2023-12-16T11:30:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:08.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.export 2023-12-16T11:30:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_4.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_4.export 2023-12-16T11:30:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_5 2023-12-16T11:30:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-16T11:30:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-16T11:30:09.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-16T11:30:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-16T11:30:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-16T11:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_5.status 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_5.status 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.status 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_5.status 2023-12-16T11:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_5.status 2023-12-16T11:30:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.status 2023-12-16T11:30:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-16T11:30:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-16T11:30:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 127 2023-12-16T11:30:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.status 2023-12-16T11:30:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.status 2023-12-16T11:30:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-16T11:30:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=127 2023-12-16T11:30:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.status 2023-12-16T11:30:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.status 2023-12-16T11:30:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-16T11:30:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 127 = '' 2023-12-16T11:30:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.status 2023-12-16T11:30:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.status 2023-12-16T11:30:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-16T11:30:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:11.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 127 = '' 2023-12-16T11:30:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.status 2023-12-16T11:30:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.status 2023-12-16T11:30:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-16T11:30:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=127 2023-12-16T11:30:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 127 = 127 2023-12-16T11:30:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-16T11:30:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:13.634 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-16T11:30:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_5': 2023-12-16T11:30:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c5e8ff38f 2023-12-16T11:30:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c5e8ff38f 2023-12-16T11:30:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:24 2023 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:24 2023 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:24 2023 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 1402ce13-80d3-4f5c-82ee-6152013b9f92 2023-12-16T11:30:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.export 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.export 2023-12-16T11:30:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.export 2023-12-16T11:30:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.export 2023-12-16T11:30:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.export 2023-12-16T11:30:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.export 2023-12-16T11:30:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_5.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_5.export 2023-12-16T11:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_6 2023-12-16T11:30:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-16T11:30:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-16T11:30:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-16T11:30:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-16T11:30:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-16T11:30:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_6.status 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_6.status 2023-12-16T11:30:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.status 2023-12-16T11:30:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_6.status 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_6.status 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.status 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-16T11:30:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 129 2023-12-16T11:30:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.status 2023-12-16T11:30:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.status 2023-12-16T11:30:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-16T11:30:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=129 2023-12-16T11:30:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.status 2023-12-16T11:30:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.status 2023-12-16T11:30:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-16T11:30:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:17.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 129 = '' 2023-12-16T11:30:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.status 2023-12-16T11:30:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.status 2023-12-16T11:30:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-16T11:30:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 129 = '' 2023-12-16T11:30:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.status 2023-12-16T11:30:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.status 2023-12-16T11:30:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-16T11:30:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=129 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 129 = 129 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_6': 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c051121c67 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c051121c67 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:31 2023 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:31 2023 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:31 2023 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: a5d4700f-a7ec-4855-83d5-735055fdfd2f 2023-12-16T11:30:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.export 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.export 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.export 2023-12-16T11:30:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.export 2023-12-16T11:30:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.export 2023-12-16T11:30:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.export 2023-12-16T11:30:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_6.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_6.export 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_7 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:21.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-16T11:30:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-12-16T11:30:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-16T11:30:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-16T11:30:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-16T11:30:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-16T11:30:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:22.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:30:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:30:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:30:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-12-16T11:30:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-16T11:30:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_7 2023-12-16T11:30:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:30:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:30:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:30:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:30:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:30:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_7 2023-12-16T11:30:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_7.status 2023-12-16T11:30:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_7.status 2023-12-16T11:30:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.status 2023-12-16T11:30:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_7.status 2023-12-16T11:30:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_7.status 2023-12-16T11:30:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.status 2023-12-16T11:30:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-16T11:30:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-16T11:30:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 131 2023-12-16T11:30:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.status 2023-12-16T11:30:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.status 2023-12-16T11:30:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-16T11:30:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=131 2023-12-16T11:30:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.status 2023-12-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.status 2023-12-16T11:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-16T11:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 131 = '' 2023-12-16T11:30:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.status 2023-12-16T11:30:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.status 2023-12-16T11:30:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-16T11:30:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 131 = '' 2023-12-16T11:30:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:24.332 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:30:24.327+0000 7f98d3a68640 -1 reset not still connected to 0x5597e220dee0 2023-12-16T11:30:24.332 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:30:24.327+0000 7f98d3a68640 -1 reset not still connected to 0x5597e22ffe10 2023-12-16T11:30:24.332 INFO:tasks.ceph.cluster1.osd.1.smithi087.stderr:2023-12-16T11:30:24.327+0000 7f98d3a68640 -1 reset not still connected to 0x5597e2375110 2023-12-16T11:30:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.status 2023-12-16T11:30:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.status 2023-12-16T11:30:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-16T11:30:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=131 2023-12-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 131 = 131 2023-12-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-16T11:30:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-16T11:30:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_7': 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12097cece4cf 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12097cece4cf 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:45 2023 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:45 2023 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:45 2023 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c054a432-d656-4791-bea1-c03a47bbbf82 2023-12-16T11:30:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.export 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.export 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.export 2023-12-16T11:30:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.export 2023-12-16T11:30:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.export 2023-12-16T11:30:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.export 2023-12-16T11:30:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_7.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_7.export 2023-12-16T11:30:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_8 2023-12-16T11:30:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-16T11:30:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-16T11:30:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-16T11:30:28.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-16T11:30:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-16T11:30:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2023-12-16T11:30:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2023-12-16T11:30:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2023-12-16T11:30:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2023-12-16T11:30:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:30:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:30:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:30:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:30:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2023-12-16T11:30:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_8.status 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_8.status 2023-12-16T11:30:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.status 2023-12-16T11:30:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_8.status 2023-12-16T11:30:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_8.status 2023-12-16T11:30:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.status 2023-12-16T11:30:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-16T11:30:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-16T11:30:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 133 2023-12-16T11:30:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.status 2023-12-16T11:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.status 2023-12-16T11:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-16T11:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=133 2023-12-16T11:30:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.status 2023-12-16T11:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.status 2023-12-16T11:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-16T11:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 133 = '' 2023-12-16T11:30:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.status 2023-12-16T11:30:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.status 2023-12-16T11:30:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-16T11:30:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 133 = '' 2023-12-16T11:30:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.status 2023-12-16T11:30:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.status 2023-12-16T11:30:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-16T11:30:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=133 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 133 = 133 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_8': 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129cfd160875 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129cfd160875 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:31.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:21:57 2023 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:21:57 2023 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:21:57 2023 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 83b3cdc9-e375-4f49-874d-7fe1521a1bba 2023-12-16T11:30:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-16T11:30:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:30:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.export 2023-12-16T11:30:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.export 2023-12-16T11:30:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.export 2023-12-16T11:30:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.export 2023-12-16T11:30:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.export 2023-12-16T11:30:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.export 2023-12-16T11:30:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_8.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_8.export 2023-12-16T11:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_9 2023-12-16T11:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-16T11:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-16T11:30:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-16T11:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-16T11:30:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-16T11:30:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-12-16T11:30:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-16T11:30:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:30:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:30:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:30:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:30:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2023-12-16T11:30:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-12-16T11:30:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:34.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_9 2023-12-16T11:30:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:30:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:30:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:30:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:30:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:30:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_9 2023-12-16T11:30:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_9.status 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_9.status 2023-12-16T11:30:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.status 2023-12-16T11:30:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_9.status 2023-12-16T11:30:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_9.status 2023-12-16T11:30:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.status 2023-12-16T11:30:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-16T11:30:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-16T11:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 135 2023-12-16T11:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.status 2023-12-16T11:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.status 2023-12-16T11:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-16T11:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=135 2023-12-16T11:30:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.status 2023-12-16T11:30:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.status 2023-12-16T11:30:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-16T11:30:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 135 = '' 2023-12-16T11:30:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.status 2023-12-16T11:30:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.status 2023-12-16T11:30:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-16T11:30:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 135 = '' 2023-12-16T11:30:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.status 2023-12-16T11:30:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.status 2023-12-16T11:30:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-16T11:30:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=135 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 135 = 135 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:37.288 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:30:37.282+0000 7fceed4d4640 -1 reset not still connected to 0x55bf264ac680 2023-12-16T11:30:37.288 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:30:37.282+0000 7fceed4d4640 -1 reset not still connected to 0x55bf27d1f380 2023-12-16T11:30:37.288 INFO:tasks.ceph.cluster1.osd.2.smithi087.stderr:2023-12-16T11:30:37.282+0000 7fceed4d4640 -1 reset not still connected to 0x55bf2a117ee0 2023-12-16T11:30:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-16T11:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_9': 2023-12-16T11:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c5b4ac5bb7 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c5b4ac5bb7 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:07 2023 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:07 2023 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:07 2023 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 73805cc2-ee84-44e2-99c5-7e848da6a3d9 2023-12-16T11:30:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-16T11:30:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:30:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.export 2023-12-16T11:30:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.export 2023-12-16T11:30:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.export 2023-12-16T11:30:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.export 2023-12-16T11:30:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.export 2023-12-16T11:30:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.export 2023-12-16T11:30:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_9.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_9.export 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_10 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:38.541 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6905695945548672) > weight (0.25) 2023-12-16T11:30:38.541 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-16T11:30:38.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-16T11:30:38.542 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-16T11:30:38.542 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-16T11:30:38.542 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:30:38.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:30:38.538+0000 7f6a7ef1d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 97256) UID: 1000 2023-12-16T11:30:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-16T11:30:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-16T11:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-16T11:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-16T11:30:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-16T11:30:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-12-16T11:30:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-12-16T11:30:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-16T11:30:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_10.status 2023-12-16T11:30:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_10.status 2023-12-16T11:30:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.status 2023-12-16T11:30:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_10.status 2023-12-16T11:30:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_10.status 2023-12-16T11:30:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.status 2023-12-16T11:30:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-16T11:30:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-16T11:30:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 137 2023-12-16T11:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.status 2023-12-16T11:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.status 2023-12-16T11:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-16T11:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=137 2023-12-16T11:30:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.status 2023-12-16T11:30:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.status 2023-12-16T11:30:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-16T11:30:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 137 = '' 2023-12-16T11:30:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.status 2023-12-16T11:30:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.status 2023-12-16T11:30:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-16T11:30:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 137 = '' 2023-12-16T11:30:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.status 2023-12-16T11:30:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.status 2023-12-16T11:30:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-16T11:30:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=137 2023-12-16T11:30:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 137 = 137 2023-12-16T11:30:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-16T11:30:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_10': 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c025612ea2 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c025612ea2 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:13 2023 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:13 2023 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:13 2023 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:43.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 23a9ee20-37a2-4d39-b844-a43578e6e5f2 2023-12-16T11:30:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-16T11:30:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:30:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.export 2023-12-16T11:30:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.export 2023-12-16T11:30:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.export 2023-12-16T11:30:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.export 2023-12-16T11:30:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.export 2023-12-16T11:30:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:44.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.export 2023-12-16T11:30:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_10.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_10.export 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_11 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:44.644 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-16T11:30:44.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-16T11:30:44.645 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-16T11:30:44.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-16T11:30:44.648 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:46.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-16T11:30:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-16T11:30:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-16T11:30:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_11.status 2023-12-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_11.status 2023-12-16T11:30:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.status 2023-12-16T11:30:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_11.status 2023-12-16T11:30:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_11.status 2023-12-16T11:30:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.status 2023-12-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-16T11:30:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 139 2023-12-16T11:30:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.status 2023-12-16T11:30:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.status 2023-12-16T11:30:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-16T11:30:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=139 2023-12-16T11:30:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.status 2023-12-16T11:30:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.status 2023-12-16T11:30:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-16T11:30:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 139 = '' 2023-12-16T11:30:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.status 2023-12-16T11:30:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.status 2023-12-16T11:30:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-16T11:30:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 139 = '' 2023-12-16T11:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.status 2023-12-16T11:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.status 2023-12-16T11:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-16T11:30:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=139 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 139 = 139 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_11': 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1209c49e7ae1 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1209c49e7ae1 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:29 2023 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:29 2023 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:29 2023 2023-12-16T11:30:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 05cc72d1-934d-49bc-8864-822da1eba303 2023-12-16T11:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.export 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.export 2023-12-16T11:30:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.export 2023-12-16T11:30:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.export 2023-12-16T11:30:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.export 2023-12-16T11:30:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.export 2023-12-16T11:30:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_11.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_11.export 2023-12-16T11:30:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_12 2023-12-16T11:30:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-16T11:30:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-16T11:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-12-16T11:30:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-16T11:30:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-16T11:30:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-16T11:30:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-16T11:30:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:30:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:30:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:30:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:30:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2023-12-16T11:30:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-16T11:30:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_12 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:30:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:30:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:30:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:30:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:30:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:30:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_12 2023-12-16T11:30:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_12.status 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_12.status 2023-12-16T11:30:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.status 2023-12-16T11:30:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_12.status 2023-12-16T11:30:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_12.status 2023-12-16T11:30:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.status 2023-12-16T11:30:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-16T11:30:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-16T11:30:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 141 2023-12-16T11:30:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.status 2023-12-16T11:30:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.status 2023-12-16T11:30:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-16T11:30:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=141 2023-12-16T11:30:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.status 2023-12-16T11:30:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.status 2023-12-16T11:30:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-16T11:30:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 141 = '' 2023-12-16T11:30:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:53.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.status 2023-12-16T11:30:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.status 2023-12-16T11:30:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-16T11:30:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 141 = '' 2023-12-16T11:30:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:30:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.status 2023-12-16T11:30:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.status 2023-12-16T11:30:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-16T11:30:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=141 2023-12-16T11:30:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 141 = 141 2023-12-16T11:30:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-16T11:30:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:30:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_12': 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129c2afcf971 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129c2afcf971 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:30:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:40 2023 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:40 2023 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:40 2023 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 81ebcd20-efb2-4bd1-b191-504c2f7127bc 2023-12-16T11:30:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-16T11:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:30:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.export 2023-12-16T11:30:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.export 2023-12-16T11:30:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.export 2023-12-16T11:30:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.export 2023-12-16T11:30:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.export 2023-12-16T11:30:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:30:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.export 2023-12-16T11:30:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_12.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_12.export 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_13 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:30:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:30:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:30:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:30:58.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:30:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:30:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:30:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:30:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-16T11:30:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-12-16T11:30:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:30:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:30:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:30:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-12-16T11:30:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:30:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:30:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:30:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:30:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-12-16T11:30:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:30:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:30:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:30:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:30:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:30:58.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_13.status 2023-12-16T11:30:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_13.status 2023-12-16T11:30:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.status 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_13.status 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_13.status 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.status 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:58.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-16T11:30:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 143 2023-12-16T11:30:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.status 2023-12-16T11:30:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:30:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.status 2023-12-16T11:30:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-16T11:30:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:30:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=143 2023-12-16T11:30:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:30:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.status 2023-12-16T11:30:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:59.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.status 2023-12-16T11:30:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-16T11:30:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 143 = '' 2023-12-16T11:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:30:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.status 2023-12-16T11:30:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:30:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:30:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:30:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.status 2023-12-16T11:30:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-16T11:30:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:30:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:30:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:30:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:30:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:30:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:30:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:30:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 143 = '' 2023-12-16T11:30:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:30:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:31:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.status 2023-12-16T11:31:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:31:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.status 2023-12-16T11:31:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-16T11:31:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=143 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 143 = 143 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:31:00.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-16T11:31:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_13': 2023-12-16T11:31:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c5377026 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c5377026 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:22:55 2023 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:22:55 2023 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:22:55 2023 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:31:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:31:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 05362fb1-bf44-4319-b368-19d9f2aa9ac9 2023-12-16T11:31:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:31:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-16T11:31:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:31:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:31:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.export 2023-12-16T11:31:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.export 2023-12-16T11:31:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.export 2023-12-16T11:31:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.export 2023-12-16T11:31:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.export 2023-12-16T11:31:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.export 2023-12-16T11:31:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_13.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_13.export 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_14 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:31:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-16T11:31:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-16T11:31:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:31:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-16T11:31:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:31:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:31:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-16T11:31:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-16T11:31:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-16T11:31:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:31:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:31:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:31:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:31:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:31:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:31:04.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_14.status 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_14.status 2023-12-16T11:31:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.status 2023-12-16T11:31:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_14.status 2023-12-16T11:31:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_14.status 2023-12-16T11:31:04.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.status 2023-12-16T11:31:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-16T11:31:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-16T11:31:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 145 2023-12-16T11:31:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.status 2023-12-16T11:31:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.status 2023-12-16T11:31:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-16T11:31:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:31:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=145 2023-12-16T11:31:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.status 2023-12-16T11:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.status 2023-12-16T11:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-16T11:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 145 = '' 2023-12-16T11:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:31:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.status 2023-12-16T11:31:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.status 2023-12-16T11:31:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-16T11:31:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 145 = '' 2023-12-16T11:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:31:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.status 2023-12-16T11:31:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.status 2023-12-16T11:31:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-16T11:31:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=145 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 145 = 145 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:31:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:07.320 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:31:07.313+0000 7f805d884640 -1 reset not still connected to 0x5597dc54c340 2023-12-16T11:31:07.320 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:31:07.313+0000 7f805d884640 -1 reset not still connected to 0x5597dd9540d0 2023-12-16T11:31:07.320 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:31:07.313+0000 7f805d884640 -1 reset not still connected to 0x5597dd9edc70 2023-12-16T11:31:07.320 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:31:07.313+0000 7f805d884640 -1 reset not still connected to 0x5597ddaae750 2023-12-16T11:31:07.320 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:31:07.313+0000 7f805d884640 -1 reset not still connected to 0x5597dde349c0 2023-12-16T11:31:07.320 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:31:07.314+0000 7f805d884640 -1 reset not still connected to 0x5597e2edc750 2023-12-16T11:31:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-16T11:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_14': 2023-12-16T11:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c041c80f59 2023-12-16T11:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c041c80f59 2023-12-16T11:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:03 2023 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:03 2023 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:03 2023 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 72b31ab9-6f6f-406d-aa42-5f6f80afc543 2023-12-16T11:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.export 2023-12-16T11:31:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.export 2023-12-16T11:31:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.export 2023-12-16T11:31:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.export 2023-12-16T11:31:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.export 2023-12-16T11:31:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.export 2023-12-16T11:31:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_14.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_14.export 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_15 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:31:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-16T11:31:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-16T11:31:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:31:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:31:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:31:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-16T11:31:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-16T11:31:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-16T11:31:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:10.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-12-16T11:31:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:31:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:31:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:31:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:31:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:31:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_15.status 2023-12-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_15.status 2023-12-16T11:31:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.status 2023-12-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_15.status 2023-12-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_15.status 2023-12-16T11:31:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.status 2023-12-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-16T11:31:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 147 2023-12-16T11:31:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.status 2023-12-16T11:31:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.status 2023-12-16T11:31:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-16T11:31:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=147 2023-12-16T11:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:31:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.status 2023-12-16T11:31:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.status 2023-12-16T11:31:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-16T11:31:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 147 = '' 2023-12-16T11:31:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:11.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:31:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.status 2023-12-16T11:31:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.status 2023-12-16T11:31:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-16T11:31:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 147 = '' 2023-12-16T11:31:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:31:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.status 2023-12-16T11:31:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.status 2023-12-16T11:31:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-16T11:31:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=147 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 147 = 147 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:31:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-16T11:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_15': 2023-12-16T11:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12091e377ca8 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12091e377ca8 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:09 2023 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:09 2023 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:09 2023 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 2bed7332-6b14-4681-8280-1b20686fbfe2 2023-12-16T11:31:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-16T11:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:31:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:31:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.export 2023-12-16T11:31:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.export 2023-12-16T11:31:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.export 2023-12-16T11:31:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.export 2023-12-16T11:31:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.export 2023-12-16T11:31:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.export 2023-12-16T11:31:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_15.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_15.export 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_16 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-16T11:31:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:31:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:31:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:31:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-16T11:31:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-16T11:31:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:31:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:31:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:31:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:31:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_16.status 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_16.status 2023-12-16T11:31:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.status 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_16.status 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_16.status 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.status 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-16T11:31:17.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 149 2023-12-16T11:31:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.status 2023-12-16T11:31:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.status 2023-12-16T11:31:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-16T11:31:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:31:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=149 2023-12-16T11:31:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:31:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.status 2023-12-16T11:31:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.status 2023-12-16T11:31:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-16T11:31:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 149 = '' 2023-12-16T11:31:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:31:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.status 2023-12-16T11:31:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.status 2023-12-16T11:31:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-16T11:31:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 149 = '' 2023-12-16T11:31:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:31:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.status 2023-12-16T11:31:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.status 2023-12-16T11:31:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-16T11:31:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=149 2023-12-16T11:31:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 149 = 149 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:31:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-16T11:31:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_16': 2023-12-16T11:31:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:31:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:31:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:31:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129c1bf82d47 2023-12-16T11:31:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129c1bf82d47 2023-12-16T11:31:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:21 2023 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:21 2023 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:21 2023 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 6981c0f3-5872-4bbd-931e-0d0c2fde30ef 2023-12-16T11:31:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.export 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.export 2023-12-16T11:31:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.export 2023-12-16T11:31:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.export 2023-12-16T11:31:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.export 2023-12-16T11:31:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.export 2023-12-16T11:31:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_16.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_16.export 2023-12-16T11:31:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:31:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_17 2023-12-16T11:31:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-16T11:31:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:31:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-16T11:31:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-16T11:31:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:31:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-16T11:31:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:31:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:31:22.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:31:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-16T11:31:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-16T11:31:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-16T11:31:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-12-16T11:31:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:31:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:31:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:31:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:31:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:31:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:31:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_17.status 2023-12-16T11:31:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_17.status 2023-12-16T11:31:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.status 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_17.status 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_17.status 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.status 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-16T11:31:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 151 2023-12-16T11:31:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.status 2023-12-16T11:31:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.status 2023-12-16T11:31:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-16T11:31:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:31:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=151 2023-12-16T11:31:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:31:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.status 2023-12-16T11:31:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.status 2023-12-16T11:31:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-16T11:31:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 151 = '' 2023-12-16T11:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:31:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.status 2023-12-16T11:31:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.status 2023-12-16T11:31:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-16T11:31:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 151 = '' 2023-12-16T11:31:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:31:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.status 2023-12-16T11:31:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.status 2023-12-16T11:31:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-16T11:31:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:24.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=151 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 151 = 151 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:31:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-16T11:31:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_17': 2023-12-16T11:31:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b8fa607fa 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b8fa607fa 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:31 2023 2023-12-16T11:31:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:31 2023 2023-12-16T11:31:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:31 2023 2023-12-16T11:31:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:31:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:31:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:31:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 476836f7-4398-4e1c-91b2-edee8e248309 2023-12-16T11:31:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.export 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.export 2023-12-16T11:31:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.export 2023-12-16T11:31:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.export 2023-12-16T11:31:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:25.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.export 2023-12-16T11:31:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.export 2023-12-16T11:31:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_17.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_17.export 2023-12-16T11:31:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:31:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_18 2023-12-16T11:31:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-16T11:31:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:31:26.557 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3956062904949139) > weight (0.25) 2023-12-16T11:31:26.557 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-16T11:31:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-16T11:31:26.558 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-16T11:31:26.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:31:26.554+0000 7f408ac32640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 105427) UID: 1000 2023-12-16T11:31:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-16T11:31:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-16T11:31:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:31:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:31:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-16T11:31:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-16T11:31:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-16T11:31:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-12-16T11:31:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-12-16T11:31:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-12-16T11:31:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-12-16T11:31:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2023-12-16T11:31:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:31:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:31:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:31:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2023-12-16T11:31:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:31:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:31:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:31:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:31:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_18.status 2023-12-16T11:31:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_18.status 2023-12-16T11:31:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.status 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_18.status 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_18.status 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.status 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-16T11:31:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 153 2023-12-16T11:31:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.status 2023-12-16T11:31:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.status 2023-12-16T11:31:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-16T11:31:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:31:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=153 2023-12-16T11:31:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:31:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.status 2023-12-16T11:31:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.status 2023-12-16T11:31:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-16T11:31:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 153 = '' 2023-12-16T11:31:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:31:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.status 2023-12-16T11:31:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.status 2023-12-16T11:31:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-16T11:31:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 153 = '' 2023-12-16T11:31:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:31:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.status 2023-12-16T11:31:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.status 2023-12-16T11:31:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-16T11:31:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=153 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 153 = 153 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:31:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_18': 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b83ec7a4eb 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b83ec7a4eb 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:31:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:41 2023 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:41 2023 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:41 2023 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c8047879-aa5e-417a-81ea-d0a79500d75f 2023-12-16T11:31:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.export 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.export 2023-12-16T11:31:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.export 2023-12-16T11:31:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.export 2023-12-16T11:31:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.export 2023-12-16T11:31:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.export 2023-12-16T11:31:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_18.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_18.export 2023-12-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_19 2023-12-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:31:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-16T11:31:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-16T11:31:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:31:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:33.559 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-16T11:31:33.559 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:31:33.559 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-16T11:31:33.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:31:33.559 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:31:33.562 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:31:33.562 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-16T11:31:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-16T11:31:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:31:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:31:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:31:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-16T11:31:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-16T11:31:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-16T11:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-16T11:31:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-16T11:31:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2023-12-16T11:31:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:31:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:31:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:31:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:31:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2023-12-16T11:31:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-16T11:31:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_19 2023-12-16T11:31:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:31:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:31:34.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:31:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_19 2023-12-16T11:31:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:31:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:31:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:31:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:31:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:31:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_19.status 2023-12-16T11:31:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_19.status 2023-12-16T11:31:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.status 2023-12-16T11:31:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_19.status 2023-12-16T11:31:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_19.status 2023-12-16T11:31:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:31:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.status 2023-12-16T11:31:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-16T11:31:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-16T11:31:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 155 2023-12-16T11:31:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.status 2023-12-16T11:31:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.status 2023-12-16T11:31:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-16T11:31:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:31:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=155 2023-12-16T11:31:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:31:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.status 2023-12-16T11:31:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.status 2023-12-16T11:31:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-16T11:31:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 155 = '' 2023-12-16T11:31:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:31:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.status 2023-12-16T11:31:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.status 2023-12-16T11:31:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-16T11:31:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:31:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:31:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:31:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:31:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:31:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 155 = '' 2023-12-16T11:31:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:31:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:31:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.status 2023-12-16T11:31:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.status 2023-12-16T11:31:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-16T11:31:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:31:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:31:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=155 2023-12-16T11:31:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 155 = 155 2023-12-16T11:31:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_19': 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b8295fea24 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b8295fea24 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:31:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:49 2023 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:49 2023 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:49 2023 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: b5372d96-8f2a-4940-b250-2dbc5ed7b604 2023-12-16T11:31:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-16T11:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.export 2023-12-16T11:31:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.export 2023-12-16T11:31:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.export 2023-12-16T11:31:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.export 2023-12-16T11:31:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.export 2023-12-16T11:31:38.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:31:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.export 2023-12-16T11:31:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_19.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_19.export 2023-12-16T11:31:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_20 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:31:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:31:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-16T11:31:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:31:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-16T11:31:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:31:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:31:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:31:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:31:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-16T11:31:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:31:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:31:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:31:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:31:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:31:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:31:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:40.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:40.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-16T11:31:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-16T11:31:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-16T11:31:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:31:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:31:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:31:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-16T11:31:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:31:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:31:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:31:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:31:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-16T11:31:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 1 2023-12-16T11:31:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-16T11:31:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-16T11:31:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-16T11:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:31:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:31:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-16T11:31:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:31:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:31:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:31:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:31:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-16T11:31:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 2 2023-12-16T11:31:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-16T11:31:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-16T11:31:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:44.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-16T11:31:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:31:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:31:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:31:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-16T11:31:44.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-16T11:31:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:31:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:31:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:31:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:31:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-16T11:31:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 4 2023-12-16T11:31:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-16T11:31:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-16T11:31:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-16T11:31:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:31:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:31:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-16T11:31:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:31:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:31:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:31:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-16T11:31:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 8 2023-12-16T11:31:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:31:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:31:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:31:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-16T11:31:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:31:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:31:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:31:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:31:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-16T11:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-16T11:31:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:31:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:31:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:31:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:31:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-16T11:31:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:31:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:31:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:31:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:31:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:31:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:31:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:31:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:31:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:31:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-16T11:31:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:31:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:31:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:31:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:31:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 8 2023-12-16T11:31:57.624 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:31:57.619+0000 7f13f7b15640 -1 reset not still connected to 0x5589bb919520 2023-12-16T11:31:57.625 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:31:57.619+0000 7f13f7b15640 -1 reset not still connected to 0x5589c1dc7520 2023-12-16T11:31:57.625 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:31:57.619+0000 7f13f7b15640 -1 reset not still connected to 0x5589e5cfaea0 2023-12-16T11:31:57.625 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:31:57.619+0000 7f13f7b15640 -1 reset not still connected to 0x5589ff0968f0 2023-12-16T11:31:57.625 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:31:57.619+0000 7f13f7b15640 -1 reset not still connected to 0x558a00074000 2023-12-16T11:31:57.625 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:31:57.621+0000 7f13f7b15640 -1 reset not still connected to 0x558a00ed5930 2023-12-16T11:32:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-16T11:32:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-16T11:32:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:32:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-16T11:32:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-16T11:32:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:32:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:32:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:32:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:32:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:05.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-12-16T11:32:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-16T11:32:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:32:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:32:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:32:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:32:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:32:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:32:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:32:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-12-16T11:32:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-16T11:32:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_20.status 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_20.status 2023-12-16T11:32:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.status 2023-12-16T11:32:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_20.status 2023-12-16T11:32:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_20.status 2023-12-16T11:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.status 2023-12-16T11:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-16T11:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-16T11:32:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 157 2023-12-16T11:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.status 2023-12-16T11:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.status 2023-12-16T11:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-16T11:32:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=157 2023-12-16T11:32:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.status 2023-12-16T11:32:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.status 2023-12-16T11:32:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-16T11:32:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 157 = '' 2023-12-16T11:32:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.status 2023-12-16T11:32:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.status 2023-12-16T11:32:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-16T11:32:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:06.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 157 = '' 2023-12-16T11:32:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:06.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.status 2023-12-16T11:32:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.status 2023-12-16T11:32:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-16T11:32:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=157 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 157 = 157 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-16T11:32:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_20': 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b8d0dba44d 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b8d0dba44d 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:23:56 2023 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:23:56 2023 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:23:56 2023 2023-12-16T11:32:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: a2f9df53-7e43-494c-88e1-9a03c24ccda8 2023-12-16T11:32:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-16T11:32:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:32:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.export 2023-12-16T11:32:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.export 2023-12-16T11:32:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.export 2023-12-16T11:32:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.export 2023-12-16T11:32:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.export 2023-12-16T11:32:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.export 2023-12-16T11:32:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_20.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_20.export 2023-12-16T11:32:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_21 2023-12-16T11:32:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-16T11:32:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:09.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-16T11:32:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-16T11:32:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:11.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:11.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-16T11:32:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-16T11:32:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:11.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-12-16T11:32:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-12-16T11:32:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-12-16T11:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:32:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:32:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-12-16T11:32:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_21.status 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_21.status 2023-12-16T11:32:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.status 2023-12-16T11:32:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_21.status 2023-12-16T11:32:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_21.status 2023-12-16T11:32:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.status 2023-12-16T11:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-16T11:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-16T11:32:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 159 2023-12-16T11:32:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.status 2023-12-16T11:32:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.status 2023-12-16T11:32:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-16T11:32:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=159 2023-12-16T11:32:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.status 2023-12-16T11:32:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.status 2023-12-16T11:32:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-16T11:32:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 159 = '' 2023-12-16T11:32:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.status 2023-12-16T11:32:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.status 2023-12-16T11:32:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-16T11:32:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 159 = '' 2023-12-16T11:32:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.status 2023-12-16T11:32:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.status 2023-12-16T11:32:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-16T11:32:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=159 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 159 = 159 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_21': 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c576bf2b29 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c576bf2b29 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:10 2023 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:10 2023 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:10 2023 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 701a286f-a667-4073-9a30-e7a68ef93705 2023-12-16T11:32:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.export 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.export 2023-12-16T11:32:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.export 2023-12-16T11:32:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.export 2023-12-16T11:32:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.export 2023-12-16T11:32:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.export 2023-12-16T11:32:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_21.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_21.export 2023-12-16T11:32:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_22 2023-12-16T11:32:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-16T11:32:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-16T11:32:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-16T11:32:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-16T11:32:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-16T11:32:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-16T11:32:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:32:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:32:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:32:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2023-12-16T11:32:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_22.status 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_22.status 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.status 2023-12-16T11:32:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_22.status 2023-12-16T11:32:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_22.status 2023-12-16T11:32:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.status 2023-12-16T11:32:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-16T11:32:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-16T11:32:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 161 2023-12-16T11:32:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.status 2023-12-16T11:32:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.status 2023-12-16T11:32:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-16T11:32:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=161 2023-12-16T11:32:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.status 2023-12-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.status 2023-12-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-16T11:32:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 161 = '' 2023-12-16T11:32:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.status 2023-12-16T11:32:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.status 2023-12-16T11:32:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-16T11:32:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 161 = '' 2023-12-16T11:32:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:18.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:19.648 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4353618977873379) > weight (0.25) 2023-12-16T11:32:19.648 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5560124597034581) > weight (0.25) 2023-12-16T11:32:19.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.34226170341458695) > weight (0.25) 2023-12-16T11:32:19.649 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.49744006334753477) > weight (0.25) 2023-12-16T11:32:19.649 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-16T11:32:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.status 2023-12-16T11:32:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.status 2023-12-16T11:32:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-16T11:32:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=161 2023-12-16T11:32:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 161 = 161 2023-12-16T11:32:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_22': 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c07b1ab52e 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c07b1ab52e 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:16 2023 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:16 2023 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:16 2023 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: de08cde6-faea-4649-8611-8d30022bde8d 2023-12-16T11:32:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.export 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.export 2023-12-16T11:32:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.export 2023-12-16T11:32:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.export 2023-12-16T11:32:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.export 2023-12-16T11:32:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.export 2023-12-16T11:32:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_22.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_22.export 2023-12-16T11:32:21.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_23 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-16T11:32:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-16T11:32:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-16T11:32:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_23.status 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_23.status 2023-12-16T11:32:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.status 2023-12-16T11:32:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_23.status 2023-12-16T11:32:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_23.status 2023-12-16T11:32:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.status 2023-12-16T11:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-16T11:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-16T11:32:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 163 2023-12-16T11:32:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.status 2023-12-16T11:32:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.status 2023-12-16T11:32:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-16T11:32:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=163 2023-12-16T11:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.status 2023-12-16T11:32:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.status 2023-12-16T11:32:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-16T11:32:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 163 = '' 2023-12-16T11:32:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.status 2023-12-16T11:32:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.status 2023-12-16T11:32:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-16T11:32:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 163 = '' 2023-12-16T11:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.status 2023-12-16T11:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.status 2023-12-16T11:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-16T11:32:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:25.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=163 2023-12-16T11:32:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 163 = 163 2023-12-16T11:32:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-16T11:32:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_23': 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129c40467f8b 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129c40467f8b 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:27 2023 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:27 2023 2023-12-16T11:32:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:27 2023 2023-12-16T11:32:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 0ff47fdb-18fc-451e-9837-b80cef74a025 2023-12-16T11:32:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.export 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.export 2023-12-16T11:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.export 2023-12-16T11:32:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.export 2023-12-16T11:32:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.export 2023-12-16T11:32:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.export 2023-12-16T11:32:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_23.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_23.export 2023-12-16T11:32:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_24 2023-12-16T11:32:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-16T11:32:27.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:27.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-16T11:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-16T11:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:28.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-16T11:32:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-16T11:32:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:29.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-16T11:32:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-16T11:32:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-12-16T11:32:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-12-16T11:32:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-12-16T11:32:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:32:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:32:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:32:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2023-12-16T11:32:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-16T11:32:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_24.status 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_24.status 2023-12-16T11:32:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.status 2023-12-16T11:32:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_24.status 2023-12-16T11:32:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_24.status 2023-12-16T11:32:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.status 2023-12-16T11:32:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-16T11:32:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-16T11:32:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 165 2023-12-16T11:32:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.status 2023-12-16T11:32:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.status 2023-12-16T11:32:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-16T11:32:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=165 2023-12-16T11:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.status 2023-12-16T11:32:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.status 2023-12-16T11:32:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-16T11:32:30.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 165 = '' 2023-12-16T11:32:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.status 2023-12-16T11:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.status 2023-12-16T11:32:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-16T11:32:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 165 = '' 2023-12-16T11:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.status 2023-12-16T11:32:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.status 2023-12-16T11:32:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-16T11:32:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=165 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 165 = 165 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-16T11:32:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_24': 2023-12-16T11:32:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b85ecd8708 2023-12-16T11:32:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b85ecd8708 2023-12-16T11:32:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:38 2023 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:38 2023 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:38 2023 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: baa6790f-87a2-4cff-9c34-1c577908fc53 2023-12-16T11:32:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-16T11:32:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:32:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.export 2023-12-16T11:32:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.export 2023-12-16T11:32:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.export 2023-12-16T11:32:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.export 2023-12-16T11:32:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.export 2023-12-16T11:32:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.export 2023-12-16T11:32:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_24.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_24.export 2023-12-16T11:32:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_25 2023-12-16T11:32:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-16T11:32:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-16T11:32:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-16T11:32:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-16T11:32:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-16T11:32:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-16T11:32:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-16T11:32:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_25.status 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_25.status 2023-12-16T11:32:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.status 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_25.status 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_25.status 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.status 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-16T11:32:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 167 2023-12-16T11:32:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.status 2023-12-16T11:32:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.status 2023-12-16T11:32:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-16T11:32:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=167 2023-12-16T11:32:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.status 2023-12-16T11:32:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.status 2023-12-16T11:32:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-16T11:32:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 167 = '' 2023-12-16T11:32:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.status 2023-12-16T11:32:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.status 2023-12-16T11:32:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-16T11:32:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 167 = '' 2023-12-16T11:32:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:37.563 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4484041749597094) > weight (0.25) 2023-12-16T11:32:37.564 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3740168988556408) > weight (0.25) 2023-12-16T11:32:37.564 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3168061229668404) > weight (0.25) 2023-12-16T11:32:37.564 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9602170431398019) > weight (0.25) 2023-12-16T11:32:37.564 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-12-16T11:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.status 2023-12-16T11:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.status 2023-12-16T11:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-16T11:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=167 2023-12-16T11:32:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 167 = 167 2023-12-16T11:32:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-16T11:32:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_25': 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c5db5d60a2 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c5db5d60a2 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:46 2023 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:46 2023 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:46 2023 2023-12-16T11:32:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 6ff4a59e-4940-43b0-92d3-556cb67e8005 2023-12-16T11:32:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-16T11:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.export 2023-12-16T11:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.export 2023-12-16T11:32:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.export 2023-12-16T11:32:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.export 2023-12-16T11:32:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.export 2023-12-16T11:32:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.export 2023-12-16T11:32:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_25.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_25.export 2023-12-16T11:32:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_26 2023-12-16T11:32:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-16T11:32:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-16T11:32:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-16T11:32:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-16T11:32:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-16T11:32:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-16T11:32:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-16T11:32:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_26.status 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_26.status 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.status 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_26.status 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_26.status 2023-12-16T11:32:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-16T11:32:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-16T11:32:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 169 2023-12-16T11:32:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.status 2023-12-16T11:32:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.status 2023-12-16T11:32:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-16T11:32:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=169 2023-12-16T11:32:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-16T11:32:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=93 2023-12-16T11:32:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 169 = 93 2023-12-16T11:32:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-16T11:32:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 169 = '' 2023-12-16T11:32:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-16T11:32:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 169 = '' 2023-12-16T11:32:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:32:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.status 2023-12-16T11:32:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-16T11:32:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=169 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 169 = 169 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-16T11:32:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_26': 2023-12-16T11:32:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c0d9f4311e 2023-12-16T11:32:46.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c0d9f4311e 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:24:59 2023 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:24:59 2023 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:24:59 2023 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c10d0adc-6101-4c60-acef-48112957bc4c 2023-12-16T11:32:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.export 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.export 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.export 2023-12-16T11:32:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.export 2023-12-16T11:32:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.export 2023-12-16T11:32:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.export 2023-12-16T11:32:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_26.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_26.export 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_27 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-16T11:32:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-16T11:32:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-16T11:32:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-12-16T11:32:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-16T11:32:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:32:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:32:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:32:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-12-16T11:32:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-16T11:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_27.status 2023-12-16T11:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_27.status 2023-12-16T11:32:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.status 2023-12-16T11:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_27.status 2023-12-16T11:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_27.status 2023-12-16T11:32:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-16T11:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-16T11:32:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 171 2023-12-16T11:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.status 2023-12-16T11:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.status 2023-12-16T11:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-16T11:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=171 2023-12-16T11:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-16T11:32:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=94 2023-12-16T11:32:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 171 = 94 2023-12-16T11:32:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-16T11:32:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 171 = '' 2023-12-16T11:32:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-16T11:32:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 171 = '' 2023-12-16T11:32:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:32:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.status 2023-12-16T11:32:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-16T11:32:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=171 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 171 = 171 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-16T11:32:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_27': 2023-12-16T11:32:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129c6b6d66ac 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129c6b6d66ac 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:13 2023 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:13 2023 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:13 2023 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 5da77513-de12-45d4-b866-435a3db39e9a 2023-12-16T11:32:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.export 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.export 2023-12-16T11:32:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.export 2023-12-16T11:32:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.export 2023-12-16T11:32:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.export 2023-12-16T11:32:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.export 2023-12-16T11:32:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_27.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_27.export 2023-12-16T11:32:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_28 2023-12-16T11:32:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-16T11:32:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:32:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-16T11:32:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-16T11:32:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:32:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:32:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:32:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-16T11:32:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:32:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:32:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:32:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:32:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:32:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:32:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:32:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-16T11:32:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-16T11:32:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-12-16T11:32:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:32:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:32:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:55.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:32:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:32:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:32:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-12-16T11:32:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-16T11:32:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:32:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:32:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:32:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:32:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2023-12-16T11:32:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-16T11:32:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:32:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:32:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:32:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:32:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:32:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:32:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:32:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:32:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2023-12-16T11:32:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:32:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:32:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:32:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:32:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:32:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:32:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:32:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-16T11:32:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_28.status 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_28.status 2023-12-16T11:32:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.status 2023-12-16T11:32:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_28.status 2023-12-16T11:32:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_28.status 2023-12-16T11:32:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:32:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.status 2023-12-16T11:32:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-16T11:32:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-16T11:32:56.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 173 2023-12-16T11:32:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.status 2023-12-16T11:32:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.status 2023-12-16T11:32:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-16T11:32:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:32:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=173 2023-12-16T11:32:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:32:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.status 2023-12-16T11:32:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.status 2023-12-16T11:32:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-16T11:32:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=95 2023-12-16T11:32:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 173 = 95 2023-12-16T11:32:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:32:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.status 2023-12-16T11:32:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.status 2023-12-16T11:32:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-16T11:32:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:57.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:32:57.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:32:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:32:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:32:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:32:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 173 = '' 2023-12-16T11:32:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:32:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.status 2023-12-16T11:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.status 2023-12-16T11:32:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-16T11:32:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:32:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=173 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 173 = 173 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:32:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:32:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-16T11:32:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_28': 2023-12-16T11:32:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:32:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:32:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b89cbfa064 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b89cbfa064 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:27 2023 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:27 2023 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:27 2023 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 89b02294-084e-47fc-a8c6-56a13899278d 2023-12-16T11:32:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.export 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.export 2023-12-16T11:32:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.export 2023-12-16T11:32:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.export 2023-12-16T11:32:59.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.export 2023-12-16T11:32:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:32:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.export 2023-12-16T11:32:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_28.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_28.export 2023-12-16T11:32:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:32:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:32:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_29 2023-12-16T11:32:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-16T11:32:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:32:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:32:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:32:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:32:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-16T11:33:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:01.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-16T11:33:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-16T11:33:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-16T11:33:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-16T11:33:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:33:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:33:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-12-16T11:33:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-16T11:33:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:33:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:33:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:33:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:33:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:33:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2023-12-16T11:33:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:01.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_29.status 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_29.status 2023-12-16T11:33:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.status 2023-12-16T11:33:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_29.status 2023-12-16T11:33:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_29.status 2023-12-16T11:33:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.status 2023-12-16T11:33:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-16T11:33:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-16T11:33:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 175 2023-12-16T11:33:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.status 2023-12-16T11:33:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.status 2023-12-16T11:33:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-16T11:33:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=175 2023-12-16T11:33:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.status 2023-12-16T11:33:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.status 2023-12-16T11:33:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-16T11:33:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 175 = '' 2023-12-16T11:33:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.status 2023-12-16T11:33:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.status 2023-12-16T11:33:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-16T11:33:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 175 = '' 2023-12-16T11:33:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.status 2023-12-16T11:33:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.status 2023-12-16T11:33:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-16T11:33:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=175 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 175 = 175 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-16T11:33:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_29': 2023-12-16T11:33:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c54dddb118 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c54dddb118 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:37 2023 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:37 2023 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:37 2023 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: fafbb98e-d219-40d4-93e6-5c734a4ed51d 2023-12-16T11:33:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.export 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.export 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.export 2023-12-16T11:33:04.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.export 2023-12-16T11:33:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.export 2023-12-16T11:33:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.export 2023-12-16T11:33:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_29.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_29.export 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_30 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-16T11:33:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:07.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-16T11:33:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-16T11:33:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-16T11:33:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-16T11:33:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-16T11:33:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_30.status 2023-12-16T11:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_30.status 2023-12-16T11:33:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.status 2023-12-16T11:33:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_30.status 2023-12-16T11:33:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_30.status 2023-12-16T11:33:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:07.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.status 2023-12-16T11:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-16T11:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-16T11:33:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 177 2023-12-16T11:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.status 2023-12-16T11:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.status 2023-12-16T11:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-16T11:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:08.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=177 2023-12-16T11:33:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.status 2023-12-16T11:33:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.status 2023-12-16T11:33:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-16T11:33:08.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:08.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:08.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 177 = '' 2023-12-16T11:33:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.status 2023-12-16T11:33:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.status 2023-12-16T11:33:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-16T11:33:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 177 = '' 2023-12-16T11:33:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:08.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.status 2023-12-16T11:33:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.status 2023-12-16T11:33:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-16T11:33:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=177 2023-12-16T11:33:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 177 = 177 2023-12-16T11:33:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-16T11:33:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-16T11:33:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_30': 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c011de628 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c011de628 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:43 2023 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:43 2023 2023-12-16T11:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:43 2023 2023-12-16T11:33:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 42e05876-1774-4fdf-a810-7063093f5060 2023-12-16T11:33:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.export 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.export 2023-12-16T11:33:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.export 2023-12-16T11:33:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.export 2023-12-16T11:33:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.export 2023-12-16T11:33:11.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.export 2023-12-16T11:33:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_30.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_30.export 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_31 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-16T11:33:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-16T11:33:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-16T11:33:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-12-16T11:33:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-12-16T11:33:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-16T11:33:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:33:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:33:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:33:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:33:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2023-12-16T11:33:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_31.status 2023-12-16T11:33:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_31.status 2023-12-16T11:33:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.status 2023-12-16T11:33:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_31.status 2023-12-16T11:33:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_31.status 2023-12-16T11:33:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.status 2023-12-16T11:33:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-16T11:33:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-16T11:33:14.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 179 2023-12-16T11:33:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.status 2023-12-16T11:33:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.status 2023-12-16T11:33:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-16T11:33:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=179 2023-12-16T11:33:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.status 2023-12-16T11:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.status 2023-12-16T11:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-16T11:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 179 = '' 2023-12-16T11:33:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.status 2023-12-16T11:33:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.status 2023-12-16T11:33:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-16T11:33:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 179 = '' 2023-12-16T11:33:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:15.502 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:33:15.496+0000 7f805d884640 -1 reset not still connected to 0x5597db647c70 2023-12-16T11:33:15.502 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:33:15.496+0000 7f805d884640 -1 reset not still connected to 0x5597db75d5f0 2023-12-16T11:33:15.502 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:33:15.496+0000 7f805d884640 -1 reset not still connected to 0x5597dbb90f70 2023-12-16T11:33:15.502 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:33:15.496+0000 7f805d884640 -1 reset not still connected to 0x5597dbf9eea0 2023-12-16T11:33:15.502 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:33:15.496+0000 7f805d884640 -1 reset not still connected to 0x5597dec12410 2023-12-16T11:33:15.502 INFO:tasks.ceph.cluster1.osd.0.smithi087.stderr:2023-12-16T11:33:15.496+0000 7f805d884640 -1 reset not still connected to 0x5597e2edcf70 2023-12-16T11:33:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.status 2023-12-16T11:33:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.status 2023-12-16T11:33:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-16T11:33:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=179 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 179 = 179 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-16T11:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_31': 2023-12-16T11:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129c92e89bf9 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129c92e89bf9 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:50 2023 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:50 2023 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:50 2023 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: c20fd292-e9fd-4021-a507-e180446b4cb3 2023-12-16T11:33:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.export 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.export 2023-12-16T11:33:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.export 2023-12-16T11:33:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.export 2023-12-16T11:33:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.export 2023-12-16T11:33:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.export 2023-12-16T11:33:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_31.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_31.export 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_32 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-16T11:33:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-16T11:33:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-16T11:33:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-16T11:33:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-12-16T11:33:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-12-16T11:33:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_32.status 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_32.status 2023-12-16T11:33:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.status 2023-12-16T11:33:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_32.status 2023-12-16T11:33:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_32.status 2023-12-16T11:33:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.status 2023-12-16T11:33:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-16T11:33:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-16T11:33:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 181 2023-12-16T11:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.status 2023-12-16T11:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.status 2023-12-16T11:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-16T11:33:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=181 2023-12-16T11:33:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.status 2023-12-16T11:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.status 2023-12-16T11:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-16T11:33:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 181 = '' 2023-12-16T11:33:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.status 2023-12-16T11:33:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.status 2023-12-16T11:33:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-16T11:33:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 181 = '' 2023-12-16T11:33:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.status 2023-12-16T11:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.status 2023-12-16T11:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-16T11:33:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=181 2023-12-16T11:33:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 181 = 181 2023-12-16T11:33:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-16T11:33:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_32': 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b8a88afe47 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b8a88afe47 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:25:57 2023 2023-12-16T11:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:25:57 2023 2023-12-16T11:33:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:25:57 2023 2023-12-16T11:33:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 5d737155-c20e-4e60-ba8a-f1c27e0ded2f 2023-12-16T11:33:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.export 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.export 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.export 2023-12-16T11:33:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.export 2023-12-16T11:33:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.export 2023-12-16T11:33:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.export 2023-12-16T11:33:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_32.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_32.export 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_33 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:23.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-16T11:33:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:25.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-16T11:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-12-16T11:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-12-16T11:33:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:25.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-12-16T11:33:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-16T11:33:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_33.status 2023-12-16T11:33:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_33.status 2023-12-16T11:33:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.status 2023-12-16T11:33:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_33.status 2023-12-16T11:33:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_33.status 2023-12-16T11:33:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.status 2023-12-16T11:33:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-16T11:33:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-16T11:33:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 183 2023-12-16T11:33:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.status 2023-12-16T11:33:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.status 2023-12-16T11:33:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-16T11:33:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=183 2023-12-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.status 2023-12-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.status 2023-12-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-16T11:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:26.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 183 = '' 2023-12-16T11:33:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.status 2023-12-16T11:33:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.status 2023-12-16T11:33:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-16T11:33:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:26.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 183 = '' 2023-12-16T11:33:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.status 2023-12-16T11:33:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.status 2023-12-16T11:33:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-16T11:33:27.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=183 2023-12-16T11:33:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 183 = 183 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-16T11:33:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_33': 2023-12-16T11:33:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c57b765197 2023-12-16T11:33:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c57b765197 2023-12-16T11:33:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:05 2023 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:05 2023 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:05 2023 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: eb7a408d-b979-4c72-9d19-d5d246a9c508 2023-12-16T11:33:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.export 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.export 2023-12-16T11:33:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.export 2023-12-16T11:33:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.export 2023-12-16T11:33:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.export 2023-12-16T11:33:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.export 2023-12-16T11:33:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_33.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_33.export 2023-12-16T11:33:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_34 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-16T11:33:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-16T11:33:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-16T11:33:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-16T11:33:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:31.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_34.status 2023-12-16T11:33:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_34.status 2023-12-16T11:33:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:31.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.status 2023-12-16T11:33:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_34.status 2023-12-16T11:33:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_34.status 2023-12-16T11:33:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-16T11:33:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-16T11:33:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 185 2023-12-16T11:33:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.status 2023-12-16T11:33:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.status 2023-12-16T11:33:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-16T11:33:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:32.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=185 2023-12-16T11:33:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-16T11:33:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 185 = '' 2023-12-16T11:33:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-16T11:33:32.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 185 = '' 2023-12-16T11:33:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-16T11:33:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 185 = '' 2023-12-16T11:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1.6 2023-12-16T11:33:34.808 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:33:34.802+0000 7f13f7b15640 -1 reset not still connected to 0x5589d3ea3380 2023-12-16T11:33:34.808 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:33:34.802+0000 7f13f7b15640 -1 reset not still connected to 0x5589eee98270 2023-12-16T11:33:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.status 2023-12-16T11:33:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-16T11:33:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=185 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 185 = 185 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-16T11:33:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_34': 2023-12-16T11:33:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c071b0637 2023-12-16T11:33:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c071b0637 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:20 2023 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:20 2023 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:20 2023 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 2c0234b3-7175-40ed-8696-d104af2ef26b 2023-12-16T11:33:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-16T11:33:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:33:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.export 2023-12-16T11:33:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.export 2023-12-16T11:33:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.export 2023-12-16T11:33:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.export 2023-12-16T11:33:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.export 2023-12-16T11:33:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.export 2023-12-16T11:33:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_34.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_34.export 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_35 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-16T11:33:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-16T11:33:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-16T11:33:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-16T11:33:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_35.status 2023-12-16T11:33:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_35.status 2023-12-16T11:33:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.status 2023-12-16T11:33:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_35.status 2023-12-16T11:33:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_35.status 2023-12-16T11:33:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.status 2023-12-16T11:33:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-16T11:33:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:38.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-16T11:33:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 187 2023-12-16T11:33:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.status 2023-12-16T11:33:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.status 2023-12-16T11:33:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-16T11:33:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=187 2023-12-16T11:33:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.status 2023-12-16T11:33:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.status 2023-12-16T11:33:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-16T11:33:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 187 = '' 2023-12-16T11:33:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.status 2023-12-16T11:33:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.status 2023-12-16T11:33:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-16T11:33:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 187 = '' 2023-12-16T11:33:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.status 2023-12-16T11:33:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.status 2023-12-16T11:33:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-16T11:33:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=187 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 187 = 187 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_35': 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 129c6660f2e5 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.129c6660f2e5 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:30 2023 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:30 2023 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:30 2023 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 9e69de8c-0a99-4a9c-9e97-10fd69295190 2023-12-16T11:33:41.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.export 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.export 2023-12-16T11:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.export 2023-12-16T11:33:41.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.export 2023-12-16T11:33:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.export 2023-12-16T11:33:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.export 2023-12-16T11:33:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_35.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_35.export 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_36 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-16T11:33:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-16T11:33:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-16T11:33:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-16T11:33:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-16T11:33:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:44.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:44.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-12-16T11:33:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-16T11:33:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2023-12-16T11:33:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:33:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:33:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2023-12-16T11:33:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-16T11:33:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:33:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:33:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:33:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2023-12-16T11:33:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_36.status 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_36.status 2023-12-16T11:33:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.status 2023-12-16T11:33:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_36.status 2023-12-16T11:33:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_36.status 2023-12-16T11:33:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.status 2023-12-16T11:33:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-16T11:33:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-16T11:33:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 189 2023-12-16T11:33:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.status 2023-12-16T11:33:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.status 2023-12-16T11:33:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-16T11:33:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=189 2023-12-16T11:33:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.status 2023-12-16T11:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.status 2023-12-16T11:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-16T11:33:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 189 = '' 2023-12-16T11:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.status 2023-12-16T11:33:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.status 2023-12-16T11:33:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-16T11:33:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 189 = '' 2023-12-16T11:33:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.status 2023-12-16T11:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.status 2023-12-16T11:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-16T11:33:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=189 2023-12-16T11:33:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 189 = 189 2023-12-16T11:33:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-16T11:33:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-16T11:33:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_36': 2023-12-16T11:33:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b8b2cb797b 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b8b2cb797b 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:44 2023 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:44 2023 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:44 2023 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 6b19399b-d04e-436d-88a4-a836895fcf94 2023-12-16T11:33:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.export 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.export 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.export 2023-12-16T11:33:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.export 2023-12-16T11:33:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.export 2023-12-16T11:33:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.export 2023-12-16T11:33:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_36.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_36.export 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_37 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-16T11:33:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-16T11:33:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-16T11:33:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-16T11:33:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:33:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:33:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:33:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-12-16T11:33:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-16T11:33:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2023-12-16T11:33:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:33:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:33:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:33:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:33:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:33:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2023-12-16T11:33:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-12-16T11:33:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:33:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:33:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2023-12-16T11:33:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:33:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:33:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:33:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:33:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:33:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2023-12-16T11:33:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_37.status 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_37.status 2023-12-16T11:33:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.status 2023-12-16T11:33:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_37.status 2023-12-16T11:33:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_37.status 2023-12-16T11:33:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.status 2023-12-16T11:33:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-16T11:33:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-16T11:33:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 191 2023-12-16T11:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.status 2023-12-16T11:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.status 2023-12-16T11:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-16T11:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=191 2023-12-16T11:33:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.status 2023-12-16T11:33:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.status 2023-12-16T11:33:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-16T11:33:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 191 = '' 2023-12-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.status 2023-12-16T11:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.status 2023-12-16T11:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-16T11:33:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 191 = '' 2023-12-16T11:33:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.status 2023-12-16T11:33:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.status 2023-12-16T11:33:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-16T11:33:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=191 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 191 = 191 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-16T11:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_37': 2023-12-16T11:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c537eebddb 2023-12-16T11:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c537eebddb 2023-12-16T11:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:26:50 2023 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:26:50 2023 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:26:50 2023 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 568c8cb4-1132-4824-acb5-5d73627a32e9 2023-12-16T11:33:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-16T11:33:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:33:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.export 2023-12-16T11:33:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.export 2023-12-16T11:33:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.export 2023-12-16T11:33:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.export 2023-12-16T11:33:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.export 2023-12-16T11:33:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.export 2023-12-16T11:33:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_37.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_37.export 2023-12-16T11:33:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:33:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_38 2023-12-16T11:33:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-16T11:33:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:33:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-16T11:33:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-16T11:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:33:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:33:56.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:33:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:33:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:33:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:33:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:33:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:33:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-16T11:33:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:33:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:33:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:33:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:33:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:33:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:33:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:33:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-16T11:33:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_38.status 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_38.status 2023-12-16T11:33:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.status 2023-12-16T11:33:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_38.status 2023-12-16T11:33:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_38.status 2023-12-16T11:33:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:33:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.status 2023-12-16T11:33:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-16T11:33:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-16T11:33:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 193 2023-12-16T11:33:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.status 2023-12-16T11:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.status 2023-12-16T11:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-16T11:33:57.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:33:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=193 2023-12-16T11:33:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:33:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.status 2023-12-16T11:33:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.status 2023-12-16T11:33:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-16T11:33:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 193 = '' 2023-12-16T11:33:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:33:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.status 2023-12-16T11:33:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.status 2023-12-16T11:33:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-16T11:33:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:33:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:33:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:33:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:33:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:33:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 193 = '' 2023-12-16T11:33:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:33:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:33:58.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7604467830914161) > weight (0.25) 2023-12-16T11:33:58.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.653058436932226) > weight (0.25) 2023-12-16T11:33:58.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8738699049381595) > weight (0.25) 2023-12-16T11:33:58.650 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.32133967074414393) > weight (0.25) 2023-12-16T11:33:58.650 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-16T11:33:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.status 2023-12-16T11:33:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.status 2023-12-16T11:33:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-16T11:33:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:33:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=193 2023-12-16T11:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 193 = 193 2023-12-16T11:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-16T11:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:33:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:33:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:33:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-16T11:33:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_38': 2023-12-16T11:33:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:33:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:33:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:33:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12c85d84ebaa 2023-12-16T11:33:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12c85d84ebaa 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:00 2023 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:00 2023 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:00 2023 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 53afa996-62cf-4e71-8df9-ce49cbffdd96 2023-12-16T11:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.export 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.export 2023-12-16T11:33:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.export 2023-12-16T11:33:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.export 2023-12-16T11:33:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:33:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.export 2023-12-16T11:34:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:00.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.export 2023-12-16T11:34:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_38.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_38.export 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_39 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-16T11:34:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-16T11:34:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-16T11:34:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-16T11:34:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:02.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-16T11:34:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-16T11:34:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_39.status 2023-12-16T11:34:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_39.status 2023-12-16T11:34:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.status 2023-12-16T11:34:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_39.status 2023-12-16T11:34:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_39.status 2023-12-16T11:34:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.status 2023-12-16T11:34:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-16T11:34:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-16T11:34:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 195 2023-12-16T11:34:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.status 2023-12-16T11:34:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.status 2023-12-16T11:34:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-12-16T11:34:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=195 2023-12-16T11:34:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.status 2023-12-16T11:34:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.status 2023-12-16T11:34:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-16T11:34:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 195 = '' 2023-12-16T11:34:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.status 2023-12-16T11:34:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.status 2023-12-16T11:34:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-16T11:34:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 195 = '' 2023-12-16T11:34:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.status 2023-12-16T11:34:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.status 2023-12-16T11:34:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-16T11:34:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=195 2023-12-16T11:34:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 195 = 195 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-12-16T11:34:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_39': 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12c8e9fc3c33 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12c8e9fc3c33 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:12 2023 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:12 2023 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:12 2023 2023-12-16T11:34:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3c59eac5-6520-4cf7-9b72-d0a11d9b9398 2023-12-16T11:34:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_39 snap 2023-12-16T11:34:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:34:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.export 2023-12-16T11:34:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.export 2023-12-16T11:34:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.export 2023-12-16T11:34:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.export 2023-12-16T11:34:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.export 2023-12-16T11:34:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.export 2023-12-16T11:34:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_39.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_39.export 2023-12-16T11:34:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_40 2023-12-16T11:34:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_40 snap 2023-12-16T11:34:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-12-16T11:34:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:07.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-12-16T11:34:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-12-16T11:34:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-12-16T11:34:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:08.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:08.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:08.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-12-16T11:34:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-12-16T11:34:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_40.status 2023-12-16T11:34:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_40.status 2023-12-16T11:34:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.status 2023-12-16T11:34:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_40.status 2023-12-16T11:34:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_40.status 2023-12-16T11:34:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.status 2023-12-16T11:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-12-16T11:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-12-16T11:34:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 197 2023-12-16T11:34:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.status 2023-12-16T11:34:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.status 2023-12-16T11:34:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-12-16T11:34:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=197 2023-12-16T11:34:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.status 2023-12-16T11:34:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.status 2023-12-16T11:34:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-16T11:34:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 197 = '' 2023-12-16T11:34:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.status 2023-12-16T11:34:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.status 2023-12-16T11:34:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-16T11:34:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 197 = '' 2023-12-16T11:34:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.status 2023-12-16T11:34:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.status 2023-12-16T11:34:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-16T11:34:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=197 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 197 = 197 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_40': 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12c86a929020 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12c86a929020 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:25 2023 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:25 2023 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:25 2023 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 05144207-c91b-49f6-a34c-04c0264e6d50 2023-12-16T11:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_40 snap 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.export 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.export 2023-12-16T11:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.export 2023-12-16T11:34:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.export 2023-12-16T11:34:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.export 2023-12-16T11:34:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.export 2023-12-16T11:34:12.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_40.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_40.export 2023-12-16T11:34:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_41 2023-12-16T11:34:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_41 snap 2023-12-16T11:34:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:13.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-12-16T11:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-12-16T11:34:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:14.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-12-16T11:34:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-16T11:34:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:34:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:34:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:34:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-12-16T11:34:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-16T11:34:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:34:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:34:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:34:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2023-12-16T11:34:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-12-16T11:34:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:34:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:34:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:34:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:34:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:34:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:34:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:34:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2023-12-16T11:34:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:14.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_41.status 2023-12-16T11:34:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_41.status 2023-12-16T11:34:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.status 2023-12-16T11:34:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_41.status 2023-12-16T11:34:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_41.status 2023-12-16T11:34:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.status 2023-12-16T11:34:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-12-16T11:34:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-12-16T11:34:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 199 2023-12-16T11:34:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.status 2023-12-16T11:34:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.status 2023-12-16T11:34:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-12-16T11:34:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=199 2023-12-16T11:34:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.status 2023-12-16T11:34:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.status 2023-12-16T11:34:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-16T11:34:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 199 = '' 2023-12-16T11:34:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.status 2023-12-16T11:34:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.status 2023-12-16T11:34:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-16T11:34:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 199 = '' 2023-12-16T11:34:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.status 2023-12-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.status 2023-12-16T11:34:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-16T11:34:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=199 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 199 = 199 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-12-16T11:34:17.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_41': 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c510d307b7 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c510d307b7 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:34 2023 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:34 2023 2023-12-16T11:34:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:34 2023 2023-12-16T11:34:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 331f3a7a-6d77-4b49-9624-c90c97974b2f 2023-12-16T11:34:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_41 snap 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.export 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.export 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.export 2023-12-16T11:34:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.export 2023-12-16T11:34:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.export 2023-12-16T11:34:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.export 2023-12-16T11:34:18.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_41.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_41.export 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_42 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_42 snap 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-12-16T11:34:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-12-16T11:34:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-16T11:34:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:34:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:34:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:34:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2023-12-16T11:34:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-16T11:34:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2023-12-16T11:34:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:34:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:34:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:34:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2023-12-16T11:34:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_42.status 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_42.status 2023-12-16T11:34:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.status 2023-12-16T11:34:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_42.status 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_42.status 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.status 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-12-16T11:34:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 201 2023-12-16T11:34:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.status 2023-12-16T11:34:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.status 2023-12-16T11:34:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-12-16T11:34:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=201 2023-12-16T11:34:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.status 2023-12-16T11:34:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.status 2023-12-16T11:34:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-16T11:34:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 201 = '' 2023-12-16T11:34:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.status 2023-12-16T11:34:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.status 2023-12-16T11:34:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-16T11:34:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 201 = '' 2023-12-16T11:34:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.status 2023-12-16T11:34:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.status 2023-12-16T11:34:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-16T11:34:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=201 2023-12-16T11:34:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 201 = 201 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:23.564 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7726200827670706) > weight (0.25) 2023-12-16T11:34:23.564 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9408350704038372) > weight (0.25) 2023-12-16T11:34:23.565 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7150811923368784) > weight (0.25) 2023-12-16T11:34:23.565 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5177304561190645) > weight (0.25) 2023-12-16T11:34:23.565 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-12-16T11:34:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-12-16T11:34:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_42': 2023-12-16T11:34:23.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c07b7c6ffb 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c07b7c6ffb 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:43 2023 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:43 2023 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:43 2023 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: f91fe4e5-2eb7-4af5-b53e-752782a7a471 2023-12-16T11:34:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_42 snap 2023-12-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.export 2023-12-16T11:34:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.export 2023-12-16T11:34:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.export 2023-12-16T11:34:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.export 2023-12-16T11:34:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.export 2023-12-16T11:34:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.export 2023-12-16T11:34:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_42.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_42.export 2023-12-16T11:34:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_43 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_43 snap 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-12-16T11:34:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-12-16T11:34:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-12-16T11:34:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-12-16T11:34:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-16T11:34:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-12-16T11:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:34:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:34:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:34:26.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:34:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-12-16T11:34:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-16T11:34:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:34:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:34:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:34:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2023-12-16T11:34:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-12-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_43.status 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_43.status 2023-12-16T11:34:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.status 2023-12-16T11:34:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_43.status 2023-12-16T11:34:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_43.status 2023-12-16T11:34:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.status 2023-12-16T11:34:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-12-16T11:34:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-12-16T11:34:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 203 2023-12-16T11:34:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.status 2023-12-16T11:34:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.status 2023-12-16T11:34:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-12-16T11:34:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=203 2023-12-16T11:34:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.status 2023-12-16T11:34:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.status 2023-12-16T11:34:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-16T11:34:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 203 = '' 2023-12-16T11:34:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.status 2023-12-16T11:34:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.status 2023-12-16T11:34:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-16T11:34:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 203 = '' 2023-12-16T11:34:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.status 2023-12-16T11:34:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.status 2023-12-16T11:34:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-16T11:34:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=203 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 203 = 203 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-12-16T11:34:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_43': 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b8b94a3d9 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b8b94a3d9 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:49 2023 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:49 2023 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:49 2023 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 90080d20-b9fb-4d39-8314-78ffc8c74551 2023-12-16T11:34:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_43 snap 2023-12-16T11:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.export 2023-12-16T11:34:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.export 2023-12-16T11:34:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.export 2023-12-16T11:34:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.export 2023-12-16T11:34:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.export 2023-12-16T11:34:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:30.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.export 2023-12-16T11:34:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_43.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_43.export 2023-12-16T11:34:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_44 2023-12-16T11:34:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_44 snap 2023-12-16T11:34:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:32.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-12-16T11:34:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-12-16T11:34:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-16T11:34:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-16T11:34:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_44.status 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_44.status 2023-12-16T11:34:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.status 2023-12-16T11:34:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_44.status 2023-12-16T11:34:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_44.status 2023-12-16T11:34:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.status 2023-12-16T11:34:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-12-16T11:34:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-12-16T11:34:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 205 2023-12-16T11:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.status 2023-12-16T11:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.status 2023-12-16T11:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-12-16T11:34:33.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=205 2023-12-16T11:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.status 2023-12-16T11:34:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.status 2023-12-16T11:34:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-16T11:34:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 205 = '' 2023-12-16T11:34:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.status 2023-12-16T11:34:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.status 2023-12-16T11:34:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-16T11:34:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 205 = '' 2023-12-16T11:34:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.status 2023-12-16T11:34:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.status 2023-12-16T11:34:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-16T11:34:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=205 2023-12-16T11:34:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 205 = 205 2023-12-16T11:34:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-12-16T11:34:34.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-12-16T11:34:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_44': 2023-12-16T11:34:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:35.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12c8c6d28086 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12c8c6d28086 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:27:56 2023 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:27:56 2023 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:27:56 2023 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 56361eef-6a64-435b-aaca-5b4920602568 2023-12-16T11:34:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_44 snap 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.export 2023-12-16T11:34:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.export 2023-12-16T11:34:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.export 2023-12-16T11:34:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.export 2023-12-16T11:34:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.export 2023-12-16T11:34:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.export 2023-12-16T11:34:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_44.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_44.export 2023-12-16T11:34:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_45 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_45 snap 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:36.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-12-16T11:34:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-12-16T11:34:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-16T11:34:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-16T11:34:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-12-16T11:34:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:34:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:34:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-12-16T11:34:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-16T11:34:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2023-12-16T11:34:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:34:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:34:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:34:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:34:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2023-12-16T11:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-16T11:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:34:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:34:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:34:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:34:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:34:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:34:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2023-12-16T11:34:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-12-16T11:34:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_45.status 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_45.status 2023-12-16T11:34:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.status 2023-12-16T11:34:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_45.status 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_45.status 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.status 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-12-16T11:34:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 207 2023-12-16T11:34:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.status 2023-12-16T11:34:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.status 2023-12-16T11:34:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-12-16T11:34:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=207 2023-12-16T11:34:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.status 2023-12-16T11:34:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.status 2023-12-16T11:34:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-16T11:34:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 207 = '' 2023-12-16T11:34:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.status 2023-12-16T11:34:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.status 2023-12-16T11:34:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-16T11:34:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 207 = '' 2023-12-16T11:34:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.status 2023-12-16T11:34:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.status 2023-12-16T11:34:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-16T11:34:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=207 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 207 = 207 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-12-16T11:34:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_45': 2023-12-16T11:34:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c5ec7d5750 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c5ec7d5750 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:09 2023 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:09 2023 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:09 2023 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3aef10c1-3b15-46e5-bf7b-1633092d93ea 2023-12-16T11:34:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_45 snap 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.export 2023-12-16T11:34:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.export 2023-12-16T11:34:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.export 2023-12-16T11:34:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.export 2023-12-16T11:34:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.export 2023-12-16T11:34:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.export 2023-12-16T11:34:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_45.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_45.export 2023-12-16T11:34:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_46 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_46 snap 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-12-16T11:34:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:44.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-12-16T11:34:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-12-16T11:34:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-16T11:34:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:34:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:34:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:34:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2023-12-16T11:34:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-12-16T11:34:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:34:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:34:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:34:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2023-12-16T11:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_46.status 2023-12-16T11:34:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_46.status 2023-12-16T11:34:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.status 2023-12-16T11:34:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_46.status 2023-12-16T11:34:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_46.status 2023-12-16T11:34:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.status 2023-12-16T11:34:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-12-16T11:34:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-12-16T11:34:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 209 2023-12-16T11:34:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.status 2023-12-16T11:34:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.status 2023-12-16T11:34:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-12-16T11:34:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:45.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=209 2023-12-16T11:34:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.status 2023-12-16T11:34:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.status 2023-12-16T11:34:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-16T11:34:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 209 = '' 2023-12-16T11:34:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.status 2023-12-16T11:34:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.status 2023-12-16T11:34:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-16T11:34:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 209 = '' 2023-12-16T11:34:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.status 2023-12-16T11:34:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.status 2023-12-16T11:34:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-16T11:34:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=209 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 209 = 209 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-12-16T11:34:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_46': 2023-12-16T11:34:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c06d973be8 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c06d973be8 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:19 2023 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:19 2023 2023-12-16T11:34:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:19 2023 2023-12-16T11:34:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: fd0fbe2d-ac26-40f9-b708-d9544fad07a5 2023-12-16T11:34:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_46 snap 2023-12-16T11:34:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:34:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.export 2023-12-16T11:34:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.export 2023-12-16T11:34:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.export 2023-12-16T11:34:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.export 2023-12-16T11:34:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.export 2023-12-16T11:34:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.export 2023-12-16T11:34:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_46.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_46.export 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_47 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_47 snap 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-12-16T11:34:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-12-16T11:34:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-16T11:34:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-16T11:34:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:34:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:34:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:34:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:34:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-16T11:34:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-16T11:34:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:34:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:34:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:34:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:34:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:34:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:34:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-12-16T11:34:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-12-16T11:34:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_47.status 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_47.status 2023-12-16T11:34:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.status 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_47.status 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_47.status 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.status 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-12-16T11:34:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 211 2023-12-16T11:34:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.status 2023-12-16T11:34:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.status 2023-12-16T11:34:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-12-16T11:34:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=211 2023-12-16T11:34:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.status 2023-12-16T11:34:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.status 2023-12-16T11:34:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-16T11:34:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 211 = '' 2023-12-16T11:34:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.status 2023-12-16T11:34:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.status 2023-12-16T11:34:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-16T11:34:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 211 = '' 2023-12-16T11:34:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.status 2023-12-16T11:34:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.status 2023-12-16T11:34:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-16T11:34:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=211 2023-12-16T11:34:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 211 = 211 2023-12-16T11:34:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-12-16T11:34:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_47': 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12b82b99c9ae 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12b82b99c9ae 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:26 2023 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:26 2023 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:26 2023 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: bca986c6-16b7-4831-979e-917d8898129e 2023-12-16T11:34:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_47 snap 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.export 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.export 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.export 2023-12-16T11:34:53.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.export 2023-12-16T11:34:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.export 2023-12-16T11:34:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:34:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.export 2023-12-16T11:34:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_47.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_47.export 2023-12-16T11:34:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:34:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_48 2023-12-16T11:34:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_48 snap 2023-12-16T11:34:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:34:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-12-16T11:34:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:34:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:34:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:34:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:34:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:34:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:34:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:34:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:34:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:34:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:34:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:34:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:34:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:34:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:34:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-12-16T11:34:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:34:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:34:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:34:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:34:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:34:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:34:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:34:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-12-16T11:34:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_48.status 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_48.status 2023-12-16T11:34:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.status 2023-12-16T11:34:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_48.status 2023-12-16T11:34:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_48.status 2023-12-16T11:34:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:34:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.status 2023-12-16T11:34:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-12-16T11:34:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-12-16T11:34:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 213 2023-12-16T11:34:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.status 2023-12-16T11:34:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:34:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.status 2023-12-16T11:34:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-12-16T11:34:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:34:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=213 2023-12-16T11:34:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:34:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.status 2023-12-16T11:34:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.status 2023-12-16T11:34:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-16T11:34:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 213 = '' 2023-12-16T11:34:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:34:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.status 2023-12-16T11:34:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.status 2023-12-16T11:34:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-16T11:34:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:34:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:34:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:34:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:34:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:34:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 213 = '' 2023-12-16T11:34:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:34:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:34:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.status 2023-12-16T11:34:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.status 2023-12-16T11:34:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-16T11:34:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:34:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:34:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=213 2023-12-16T11:34:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 213 = 213 2023-12-16T11:34:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:34:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:34:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-12-16T11:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_48': 2023-12-16T11:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 12c8f662f97 2023-12-16T11:34:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.12c8f662f97 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:32 2023 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:32 2023 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:32 2023 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 0133838f-72ed-4a6e-b784-f5eb3197806a 2023-12-16T11:34:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_48 snap 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.export 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.export 2023-12-16T11:34:59.825 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.export 2023-12-16T11:34:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.export 2023-12-16T11:35:00.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:35:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.export 2023-12-16T11:35:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:35:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.export 2023-12-16T11:35:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_48.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_48.export 2023-12-16T11:35:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_49 2023-12-16T11:35:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_49 snap 2023-12-16T11:35:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:35:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:35:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-12-16T11:35:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:35:01.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:35:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-12-16T11:35:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:35:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:35:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:35:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:35:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:35:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:35:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:35:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-12-16T11:35:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:35:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:35:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:35:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:35:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:35:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:35:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:35:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-12-16T11:35:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-16T11:35:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:35:02.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:35:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:35:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:35:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:35:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:35:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:35:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:35:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:35:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-12-16T11:35:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-16T11:35:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:35:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:35:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:35:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:35:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:35:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:35:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:35:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:35:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2023-12-16T11:35:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-16T11:35:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:35:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:3 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:3 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:35:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=3 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 3 = cluster1:3 ']' 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=3 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:3 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:3 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=3 2023-12-16T11:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 3 = cluster1:3 ']' 2023-12-16T11:35:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:35:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:35:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=3 2023-12-16T11:35:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=3 2023-12-16T11:35:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-12-16T11:35:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:35:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:35:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-16T11:35:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2023-12-16T11:35:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:35:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:35:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:35:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:35:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_49.status 2023-12-16T11:35:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_49.status 2023-12-16T11:35:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:35:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.status 2023-12-16T11:35:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_49.status 2023-12-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_49.status 2023-12-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.status 2023-12-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-12-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-12-16T11:35:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 215 2023-12-16T11:35:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.status 2023-12-16T11:35:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.status 2023-12-16T11:35:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-12-16T11:35:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:35:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=215 2023-12-16T11:35:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:35:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:35:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:35:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.status 2023-12-16T11:35:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.status 2023-12-16T11:35:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-16T11:35:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:35:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:35:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:35:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:35:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 215 = '' 2023-12-16T11:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:35:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:35:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.status 2023-12-16T11:35:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.status 2023-12-16T11:35:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-16T11:35:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:35:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:35:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:35:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:35:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:35:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:35:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 215 = '' 2023-12-16T11:35:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:35:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:35:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.status 2023-12-16T11:35:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.status 2023-12-16T11:35:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-16T11:35:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:35:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=215 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 215 = 215 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:35:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:35:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_49': 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 10c57d89be44 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.10c57d89be44 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:35:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:41 2023 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:41 2023 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:41 2023 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: fc94afcb-ef75-4403-83fc-5aacc271b450 2023-12-16T11:35:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_49 snap 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.export 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.export 2023-12-16T11:35:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.export 2023-12-16T11:35:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.export 2023-12-16T11:35:06.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:35:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.export 2023-12-16T11:35:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:35:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.export 2023-12-16T11:35:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_49.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_49.export 2023-12-16T11:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_50 2023-12-16T11:35:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_snap cluster2 mirror image_50 snap 2023-12-16T11:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:35:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-12-16T11:35:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=started 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:35:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:35:07.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.42483547338187766) > weight (0.25) 2023-12-16T11:35:07.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8242020874826719) > weight (0.25) 2023-12-16T11:35:07.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6981404118779239) > weight (0.25) 2023-12-16T11:35:07.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.717804798224157) > weight (0.25) 2023-12-16T11:35:07.651 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-12-16T11:35:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=started 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_result 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=stopped 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_instance=cluster1 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -i state 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:35:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance=cluster1 2023-12-16T11:35:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local loop_instance 2023-12-16T11:35:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-12-16T11:35:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sleep 0 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ seq 0 3 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:0 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:0 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:35:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 0 = cluster1:0 ']' 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=0 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:0 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:0 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=0 2023-12-16T11:35:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 0 = cluster1:0 ']' 2023-12-16T11:35:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:35:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:35:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=0 2023-12-16T11:35:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=0 2023-12-16T11:35:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-12-16T11:35:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:35:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:35:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-16T11:35:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-12-16T11:35:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-16T11:35:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:35:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:1 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:1 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:35:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 1 = cluster1:1 ']' 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:1 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:35:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=1 2023-12-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 1 = cluster1:1 ']' 2023-12-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=1 2023-12-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=1 2023-12-16T11:35:08.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-12-16T11:35:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:35:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:35:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-16T11:35:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-12-16T11:35:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-12-16T11:35:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd mirror status : get status for rbd mirror 2023-12-16T11:35:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:admin_socket: invalid command 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1:2 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ shift 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local val=cluster1:2 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster_var_name=cluster 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local instance_var_name=instance 2023-12-16T11:35:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=2 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ '[' 2 = cluster1:2 ']' 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval cluster=cluster1 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ eval instance=2 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local local_cluster=cluster1:2 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster=cluster1 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local val=cluster1:2 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local cluster_var_name=local_cluster 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ local instance_var_name=instance 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ cluster=cluster1 2023-12-16T11:35:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ instance=2 2023-12-16T11:35:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ '[' 2 = cluster1:2 ']' 2023-12-16T11:35:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval local_cluster=cluster1 2023-12-16T11:35:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ local_cluster=cluster1 2023-12-16T11:35:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ eval instance=2 2023-12-16T11:35:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ instance=2 2023-12-16T11:35:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-12-16T11:35:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:35:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:35:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-16T11:35:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2023-12-16T11:35:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ return 0 2023-12-16T11:35:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ status_result=' "state": "Replaying"' 2023-12-16T11:35:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo ' "state": "Replaying"' 2023-12-16T11:35:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ grep -i Replaying 2023-12-16T11:35:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: "state": "Replaying" 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=started 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test started = started 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = journal ']' 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' snapshot = snapshot ']' 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_cluster=cluster1 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster2-mirror-image_50.status 2023-12-16T11:35:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster2-mirror-image_50.status 2023-12-16T11:35:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:35:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local status_log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.status 2023-12-16T11:35:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ mkfname cluster1-mirror-image_50.status 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ echo cluster1-mirror-image_50.status 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -e 's|[/ ]|_|g' 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local local_status_log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.status 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-12-16T11:35:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Snapshot ID: 217 2023-12-16T11:35:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.status 2023-12-16T11:35:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.status 2023-12-16T11:35:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-12-16T11:35:09.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-12-16T11:35:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snapshot_id=217 2023-12-16T11:35:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:35:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:35:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.2 2023-12-16T11:35:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.status 2023-12-16T11:35:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.status 2023-12-16T11:35:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-16T11:35:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:35:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:35:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:35:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:35:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:35:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:35:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 217 = '' 2023-12-16T11:35:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:35:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.4 2023-12-16T11:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.status 2023-12-16T11:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.status 2023-12-16T11:35:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-16T11:35:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:35:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:35:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:-:1.1: Document is empty 2023-12-16T11:35:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: 2023-12-16T11:35:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:^ 2023-12-16T11:35:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id= 2023-12-16T11:35:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 217 = '' 2023-12-16T11:35:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-12-16T11:35:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.8 2023-12-16T11:35:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.status 2023-12-16T11:35:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local log=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.status 2023-12-16T11:35:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-16T11:35:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-12-16T11:35:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-16T11:35:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local primary_snapshot_id=217 2023-12-16T11:35:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 217 = 217 2023-12-16T11:35:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-12-16T11:35:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:35:11.794 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-12-16T11:35:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_50': 2023-12-16T11:35:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 128 MiB in 32 objects 2023-12-16T11:35:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:35:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 2 2023-12-16T11:35:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 11c0af9056c6 2023-12-16T11:35:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.11c0af9056c6 2023-12-16T11:35:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering, non-primary 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:28:55 2023 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:28:55 2023 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:28:55 2023 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: protected: False 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 0e617b2e-921c-479c-a203-4d86e513e03e 2023-12-16T11:35:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: false 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ compare_image_snaps mirror image_50 snap 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap_name=snap 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local ret=0 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local rmt_export=/tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.export 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local loc_export=/tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.export 2023-12-16T11:35:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.export 2023-12-16T11:35:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.export 2023-12-16T11:35:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:35:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.export 2023-12-16T11:35:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-16T11:35:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cmp /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.export 2023-12-16T11:35:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -f /tmp/tmp.jvOAE3Umds/cluster2-mirror-image_50.export /tmp/tmp.jvOAE3Umds/cluster1-mirror-image_50.export 2023-12-16T11:35:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ testlog 'TEST: delete many images' 2023-12-16T11:35:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee -a /tmp/tmp.jvOAE3Umds/rbd-mirror.test.log 2023-12-16T11:35:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ date '+%F %T' 2023-12-16T11:35:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo 2023-12-16 11:35:12 TEST: delete many images 2023-12-16T11:35:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:2023-12-16 11:35:12 TEST: delete many images 2023-12-16T11:35:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 50 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_1 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-12-16T11:35:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-16T11:35:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:35:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_1 2023-12-16T11:35:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:35:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-16T11:35:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-16T11:35:13.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:13.342+0000 7fee8ccca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc636f16c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:13.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:13.342+0000 7fee92cd6640 -1 rbd::mirror::ImageReplayer: 0x55dc626b1400 [3/3fc80ffb-aa2e-47fa-aeab-e20034fa097b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_2 2023-12-16T11:35:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-12-16T11:35:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:35:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-12-16T11:35:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-16T11:35:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:35:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_2 2023-12-16T11:35:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:35:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-16T11:35:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-16T11:35:15.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:15.408+0000 7fee8ccca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc636f1860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:15.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:15.408+0000 7fee92cd6640 -1 rbd::mirror::ImageReplayer: 0x55dc626b1680 [3/5c88cdc9-a15c-46ab-8224-c40a8d9af053] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_3 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-12-16T11:35:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-16T11:35:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:35:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_3 2023-12-16T11:35:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:35:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-16T11:35:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-16T11:35:17.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:17.420+0000 7fee8c4c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc63711040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:17.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:17.420+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc626b0280 [3/3e35d21e-975d-43a4-a669-8eaeaf673ca7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_4 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-12-16T11:35:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-16T11:35:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:35:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_4 2023-12-16T11:35:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:35:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-16T11:35:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-16T11:35:19.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:19.395+0000 7fee8c4c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc636f1a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:19.401 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:19.395+0000 7fee92cd6640 -1 rbd::mirror::ImageReplayer: 0x55dc626b0500 [3/f9aaccb8-5e33-401a-b44f-3e6c030d0568] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_5 2023-12-16T11:35:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-12-16T11:35:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:35:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-12-16T11:35:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-16T11:35:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:35:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_5 2023-12-16T11:35:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:35:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-16T11:35:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-16T11:35:21.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:35:21.578+0000 7f5cb7b3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac21860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:21.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:35:21.578+0000 7f5cbe348640 -1 rbd::mirror::ImageReplayer: 0x555f9a96a500 [3/1402ce13-80d3-4f5c-82ee-6152013b9f92] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_6 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-12-16T11:35:23.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-16T11:35:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_6 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:35:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-16T11:35:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-16T11:35:23.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:35:23.218+0000 7f5cb7b3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac21a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:23.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:35:23.218+0000 7f5cbe348640 -1 rbd::mirror::ImageReplayer: 0x555f9a96af00 [3/a5d4700f-a7ec-4855-83d5-735055fdfd2f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_7 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-12-16T11:35:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-16T11:35:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:35:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_7 2023-12-16T11:35:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:35:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-16T11:35:25.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-16T11:35:25.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:25.217+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f234680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:25.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:25.217+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354ed1a500 [3/c054a432-d656-4791-bea1-c03a47bbbf82] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_8 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-12-16T11:35:26.566 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9896066286458453) > weight (0.25) 2023-12-16T11:35:26.566 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-16T11:35:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-16T11:35:26.567 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-16T11:35:26.567 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-16T11:35:26.567 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:35:26.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:35:26.562+0000 7f374d2e3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 109769) UID: 1000 2023-12-16T11:35:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-16T11:35:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:35:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_8 2023-12-16T11:35:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:35:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-16T11:35:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-16T11:35:27.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:27.224+0000 7f28a3cd7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563311434820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:27.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:27.224+0000 7f28aace5640 -1 rbd::mirror::ImageReplayer: 0x56331128e780 [3/83b3cdc9-e375-4f49-874d-7fe1521a1bba] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_9 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-12-16T11:35:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-16T11:35:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:35:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_9 2023-12-16T11:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:35:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-16T11:35:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-16T11:35:29.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:35:29.188+0000 7ff4c8ce4640 -1 rbd::mirror::ImageMap: 0x558f34605440 handle_peer_ack_remove: failed to remove global_image_id=73805cc2-ee84-44e2-99c5-7e848da6a3d9 2023-12-16T11:35:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_10 2023-12-16T11:35:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-12-16T11:35:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:35:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-12-16T11:35:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-16T11:35:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:35:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_10 2023-12-16T11:35:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:35:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-16T11:35:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-16T11:35:31.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:35:31.207+0000 7ff4c94e5640 -1 rbd::mirror::ImageMap: 0x558f34605440 handle_peer_ack_remove: failed to remove global_image_id=23a9ee20-37a2-4d39-b844-a43578e6e5f2 2023-12-16T11:35:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_11 2023-12-16T11:35:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-12-16T11:35:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:35:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-12-16T11:35:32.669 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-16T11:35:32.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:35:32.669 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:35:32.672 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:35:32.672 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-12-16T11:35:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_11 2023-12-16T11:35:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:35:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_11 2023-12-16T11:35:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:35:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-12-16T11:35:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-12-16T11:35:33.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:35:33.171+0000 7ff4c94e5640 -1 rbd::mirror::ImageMap: 0x558f34605440 handle_peer_ack_remove: failed to remove global_image_id=05cc72d1-934d-49bc-8864-822da1eba303 2023-12-16T11:35:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_12 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-12-16T11:35:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_12 2023-12-16T11:35:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:35:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_12 2023-12-16T11:35:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:35:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-12-16T11:35:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-12-16T11:35:35.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:35.262+0000 7f28a44d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563311241040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:35.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:35.262+0000 7f28aace5640 -1 rbd::mirror::ImageReplayer: 0x56331128f680 [3/81ebcd20-efb2-4bd1-b191-504c2f7127bc] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_13 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-12-16T11:35:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_13 2023-12-16T11:35:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:35:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_13 2023-12-16T11:35:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:35:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-12-16T11:35:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-12-16T11:35:37.245 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:35:37.239+0000 7ff4c94e5640 -1 rbd::mirror::ImageMap: 0x558f34605440 handle_peer_ack_remove: failed to remove global_image_id=05362fb1-bf44-4319-b368-19d9f2aa9ac9 2023-12-16T11:35:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_14 2023-12-16T11:35:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-12-16T11:35:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:35:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-12-16T11:35:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_14 2023-12-16T11:35:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:35:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_14 2023-12-16T11:35:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:35:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-12-16T11:35:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-12-16T11:35:39.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:35:39.257+0000 7ff4c94e5640 -1 rbd::mirror::ImageMap: 0x558f34605440 handle_peer_ack_remove: failed to remove global_image_id=72b31ab9-6f6f-406d-aa42-5f6f80afc543 2023-12-16T11:35:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_15 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-12-16T11:35:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_15 2023-12-16T11:35:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:35:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_15 2023-12-16T11:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:35:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-12-16T11:35:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-12-16T11:35:41.269 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:35:41.264+0000 7ff4c94e5640 -1 rbd::mirror::ImageMap: 0x558f34605440 handle_peer_ack_remove: failed to remove global_image_id=2bed7332-6b14-4681-8280-1b20686fbfe2 2023-12-16T11:35:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_16 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-12-16T11:35:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_16 2023-12-16T11:35:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:35:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_16 2023-12-16T11:35:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:35:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-12-16T11:35:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-12-16T11:35:43.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:43.416+0000 7f28a44d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563311241380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:43.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:43.416+0000 7f28aace5640 -1 rbd::mirror::ImageReplayer: 0x56331128e500 [3/6981c0f3-5872-4bbd-931e-0d0c2fde30ef] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_17 2023-12-16T11:35:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-12-16T11:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-12-16T11:35:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_17 2023-12-16T11:35:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:35:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_17 2023-12-16T11:35:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:35:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-12-16T11:35:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-12-16T11:35:45.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:45.334+0000 7fee8ccca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc6373f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:45.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:35:45.334+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc626b0f00 [3/476836f7-4398-4e1c-91b2-edee8e248309] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_18 2023-12-16T11:35:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-12-16T11:35:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:35:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-12-16T11:35:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_18 2023-12-16T11:35:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:35:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_18 2023-12-16T11:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:35:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-12-16T11:35:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-12-16T11:35:47.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:47.333+0000 7f1c7b6a9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f235ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:47.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:47.333+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354ed1ac80 [3/c8047879-aa5e-417a-81ea-d0a79500d75f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:48.249 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:35:48.243+0000 7ff3e93a3640 -1 reset not still connected to 0x564aa03e35f0 2023-12-16T11:35:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_19 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-12-16T11:35:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_19 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_19 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:35:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-12-16T11:35:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-12-16T11:35:49.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:49.384+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f235d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:49.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:49.384+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354ed1aa00 [3/b5372d96-8f2a-4940-b250-2dbc5ed7b604] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_20 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-12-16T11:35:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_20 2023-12-16T11:35:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:35:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_20 2023-12-16T11:35:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:35:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-12-16T11:35:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-12-16T11:35:51.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:51.372+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354ec14ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:51.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:51.372+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354ed1af00 [3/a2f9df53-7e43-494c-88e1-9a03c24ccda8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_21 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-12-16T11:35:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_21 2023-12-16T11:35:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:35:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_21 2023-12-16T11:35:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:35:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-12-16T11:35:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-12-16T11:35:53.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:53.227+0000 7f28a44d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5633108376c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:53.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:35:53.227+0000 7f28aa4e4640 -1 rbd::mirror::ImageReplayer: 0x563310fcc000 [3/701a286f-a667-4073-9a30-e7a68ef93705] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_22 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-12-16T11:35:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_22 2023-12-16T11:35:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:35:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_22 2023-12-16T11:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-12-16T11:35:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-12-16T11:35:55.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:55.382+0000 7f1c7b6a9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f1f2d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:55.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:55.382+0000 7f1c826b7640 -1 rbd::mirror::ImageReplayer: 0x56354eec4c80 [3/de08cde6-faea-4649-8611-8d30022bde8d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_23 2023-12-16T11:35:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-12-16T11:35:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:35:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-12-16T11:35:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_23 2023-12-16T11:35:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:35:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_23 2023-12-16T11:35:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:35:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-12-16T11:35:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-12-16T11:35:57.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:35:57.389+0000 7f5cb7b3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac20820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:57.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:35:57.389+0000 7f5cbeb49640 -1 rbd::mirror::ImageReplayer: 0x555f9a96b680 [3/0ff47fdb-18fc-451e-9837-b80cef74a025] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:35:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:35:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:35:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_24 2023-12-16T11:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-12-16T11:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:35:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-12-16T11:35:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:35:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_24 2023-12-16T11:35:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:35:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:35:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:35:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_24 2023-12-16T11:35:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:35:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:35:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:35:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-12-16T11:35:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-12-16T11:35:59.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:59.409+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f177ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:35:59.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:35:59.409+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354ed1b400 [3/baa6790f-87a2-4cff-9c34-1c577908fc53] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_25 2023-12-16T11:36:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-12-16T11:36:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:36:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-12-16T11:36:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_25 2023-12-16T11:36:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:36:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_25 2023-12-16T11:36:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:36:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-12-16T11:36:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-12-16T11:36:01.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:01.386+0000 7f28a44d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563310837520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:01.391 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:01.386+0000 7f28aa4e4640 -1 rbd::mirror::ImageReplayer: 0x56331128fb80 [3/6ff4a59e-4940-43b0-92d3-556cb67e8005] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_26 2023-12-16T11:36:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-12-16T11:36:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:36:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-12-16T11:36:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_26 2023-12-16T11:36:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:36:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_26 2023-12-16T11:36:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:36:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-12-16T11:36:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-12-16T11:36:03.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:03.217+0000 7f28a44d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5633114c04e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:03.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:03.217+0000 7f28aa4e4640 -1 rbd::mirror::ImageReplayer: 0x563310fcca00 [3/c10d0adc-6101-4c60-acef-48112957bc4c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_27 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:04.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-12-16T11:36:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_27 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_27 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:36:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-12-16T11:36:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-12-16T11:36:05.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:05.231+0000 7f5cb833c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac6c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:05.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:05.231+0000 7f5cbe348640 -1 rbd::mirror::ImageReplayer: 0x555f9a96b900 [3/5da77513-de12-45d4-b866-435a3db39e9a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_28 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:36:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-12-16T11:36:07.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_28 2023-12-16T11:36:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:36:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_28 2023-12-16T11:36:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:36:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-12-16T11:36:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-12-16T11:36:07.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:07.233+0000 7f28a3cd7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5633114c0820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:07.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:07.233+0000 7f28aa4e4640 -1 rbd::mirror::ImageReplayer: 0x56331128ea00 [3/89b02294-084e-47fc-a8c6-56a13899278d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_29 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-12-16T11:36:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_29 2023-12-16T11:36:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:36:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_29 2023-12-16T11:36:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:36:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-12-16T11:36:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-12-16T11:36:09.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:09.249+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f1f2ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:09.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:09.249+0000 7f1c826b7640 -1 rbd::mirror::ImageReplayer: 0x56354eec4f00 [3/fafbb98e-d219-40d4-93e6-5c734a4ed51d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_30 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:36:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:10.210 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-12-16T11:36:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_30 2023-12-16T11:36:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:36:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_30 2023-12-16T11:36:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:36:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-12-16T11:36:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-12-16T11:36:11.257 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:11.253+0000 7f28a44d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5633114c1040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:11.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:11.253+0000 7f28aa4e4640 -1 rbd::mirror::ImageReplayer: 0x56331128f180 [3/42e05876-1774-4fdf-a810-7063093f5060] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_31 2023-12-16T11:36:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-12-16T11:36:12.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:36:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-12-16T11:36:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_31 2023-12-16T11:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_31 2023-12-16T11:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:36:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-12-16T11:36:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-12-16T11:36:13.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:13.236+0000 7f5cb833c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac20b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:13.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:13.236+0000 7f5cbeb49640 -1 rbd::mirror::ImageReplayer: 0x555f9a720a00 [3/c20fd292-e9fd-4021-a507-e180446b4cb3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_32 2023-12-16T11:36:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-12-16T11:36:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:36:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-12-16T11:36:15.104 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:15.114 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_32 2023-12-16T11:36:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:36:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_32 2023-12-16T11:36:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:36:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-12-16T11:36:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-12-16T11:36:15.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:15.254+0000 7f1c7b6a9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f2341a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:15.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:15.254+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354ed1b680 [3/5d737155-c20e-4e60-ba8a-f1c27e0ded2f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_33 2023-12-16T11:36:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-12-16T11:36:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:36:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-12-16T11:36:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_33 2023-12-16T11:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_33 2023-12-16T11:36:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:36:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-12-16T11:36:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-12-16T11:36:17.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:17.467+0000 7fee8c4c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc6287d6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:17.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:17.468+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc626b1900 [3/eb7a408d-b979-4c72-9d19-d5d246a9c508] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:18.212 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:36:18.207+0000 7ff3e93a3640 -1 reset not still connected to 0x564aa2329040 2023-12-16T11:36:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_34 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-12-16T11:36:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_34 2023-12-16T11:36:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:36:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_34 2023-12-16T11:36:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:36:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-12-16T11:36:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-12-16T11:36:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:19.278+0000 7f28a44d8640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563311465a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:19.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:19.278+0000 7f28aace5640 -1 rbd::mirror::ImageReplayer: 0x56331128ef00 [3/2c0234b3-7175-40ed-8696-d104af2ef26b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_35 2023-12-16T11:36:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-12-16T11:36:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:36:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-12-16T11:36:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_35 2023-12-16T11:36:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:36:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_35 2023-12-16T11:36:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:36:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-12-16T11:36:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-12-16T11:36:21.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:21.307+0000 7f5cb833c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac20d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:21.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:21.307+0000 7f5cbeb49640 -1 rbd::mirror::ImageReplayer: 0x555f9a720000 [3/9e69de8c-0a99-4a9c-9e97-10fd69295190] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_36 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-12-16T11:36:23.083 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_36 2023-12-16T11:36:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:36:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_36 2023-12-16T11:36:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:36:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-12-16T11:36:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-12-16T11:36:23.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:23.237+0000 7fee8ccca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc6287d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:23.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:23.237+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc626b0780 [3/6b19399b-d04e-436d-88a4-a836895fcf94] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:24.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_37 2023-12-16T11:36:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-12-16T11:36:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:36:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-12-16T11:36:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_37 2023-12-16T11:36:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:36:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_37 2023-12-16T11:36:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:36:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-12-16T11:36:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-12-16T11:36:25.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:25.313+0000 7fee8ccca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc636f01a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:25.318 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:25.313+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc626b0a00 [3/568c8cb4-1132-4824-acb5-5d73627a32e9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_38 2023-12-16T11:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-12-16T11:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-12-16T11:36:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_38 2023-12-16T11:36:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:36:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_38 2023-12-16T11:36:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:36:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-12-16T11:36:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-12-16T11:36:27.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:27.337+0000 7f5cb7b3b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac20ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:27.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:27.337+0000 7f5cbeb49640 -1 rbd::mirror::ImageReplayer: 0x555f9a96bb80 [3/53afa996-62cf-4e71-8df9-ce49cbffdd96] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:28.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_39 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-12-16T11:36:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_39 2023-12-16T11:36:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:36:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_39 2023-12-16T11:36:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:36:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-12-16T11:36:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-12-16T11:36:29.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:29.331+0000 7f1c7b6a9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f234b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:29.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:29.331+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354eec4000 [3/3c59eac5-6520-4cf7-9b72-d0a11d9b9398] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_40 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-12-16T11:36:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_40 2023-12-16T11:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_40 2023-12-16T11:36:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:36:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-12-16T11:36:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-12-16T11:36:31.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:31.534+0000 7fee8ccca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc6374f520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:31.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:31.534+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc626b1b80 [3/05144207-c91b-49f6-a34c-04c0264e6d50] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_41 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:32.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-12-16T11:36:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_41 2023-12-16T11:36:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:36:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_41 2023-12-16T11:36:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:36:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-12-16T11:36:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-12-16T11:36:33.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:33.229+0000 7f28a3cd7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56331155d380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:33.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:33.229+0000 7f28aace5640 -1 rbd::mirror::ImageReplayer: 0x56331128ec80 [3/331f3a7a-6d77-4b49-9624-c90c97974b2f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_42 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-12-16T11:36:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_42 2023-12-16T11:36:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:36:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_42 2023-12-16T11:36:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:36:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-12-16T11:36:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-12-16T11:36:35.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:35.343+0000 7f28a3cd7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56331155d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:35.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:35.343+0000 7f28aace5640 -1 rbd::mirror::ImageReplayer: 0x563310fcc280 [3/f91fe4e5-2eb7-4af5-b53e-752782a7a471] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_43 2023-12-16T11:36:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-12-16T11:36:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:36:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-12-16T11:36:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_43 2023-12-16T11:36:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:36:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_43 2023-12-16T11:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:36:37.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-12-16T11:36:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-12-16T11:36:37.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:37.371+0000 7f1c7b6a9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f1f36c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:37.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:37.371+0000 7f1c826b7640 -1 rbd::mirror::ImageReplayer: 0x56354eec4280 [3/90080d20-b9fb-4d39-8314-78ffc8c74551] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:38.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:38.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_44 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-12-16T11:36:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_44 2023-12-16T11:36:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:36:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_44 2023-12-16T11:36:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:36:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-12-16T11:36:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-12-16T11:36:39.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:39.363+0000 7fee8c4c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc6374f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:39.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:39.363+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc627d4000 [3/56361eef-6a64-435b-aaca-5b4920602568] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_45 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-12-16T11:36:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_45 2023-12-16T11:36:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:36:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_45 2023-12-16T11:36:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:36:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-12-16T11:36:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-12-16T11:36:41.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:41.390+0000 7f28a3cd7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56331155d6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:41.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:36:41.390+0000 7f28aace5640 -1 rbd::mirror::ImageReplayer: 0x56331128f400 [3/3aef10c1-3b15-46e5-bf7b-1633092d93ea] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_46 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-12-16T11:36:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_46 2023-12-16T11:36:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:36:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_46 2023-12-16T11:36:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:36:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-12-16T11:36:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-12-16T11:36:43.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:43.258+0000 7f1c7b6a9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f163ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:43.262 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:43.258+0000 7f1c826b7640 -1 rbd::mirror::ImageReplayer: 0x56354ecccf00 [3/fd0fbe2d-ac26-40f9-b708-d9544fad07a5] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:44.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_47 2023-12-16T11:36:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-12-16T11:36:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:36:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-12-16T11:36:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_47 2023-12-16T11:36:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:36:45.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_47 2023-12-16T11:36:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:45.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:36:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-12-16T11:36:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-12-16T11:36:45.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:45.383+0000 7f1c7b6a9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f235520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:45.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:36:45.383+0000 7f1c81eb6640 -1 rbd::mirror::ImageReplayer: 0x56354eec4a00 [3/bca986c6-16b7-4831-979e-917d8898129e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_48 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-12-16T11:36:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_48 2023-12-16T11:36:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:36:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_48 2023-12-16T11:36:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:36:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-12-16T11:36:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-12-16T11:36:47.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:47.399+0000 7fee8c4c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc6374e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:47.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:47.399+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc627d4280 [3/0133838f-72ed-4a6e-b784-f5eb3197806a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_49 2023-12-16T11:36:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-12-16T11:36:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:36:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-12-16T11:36:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_49 2023-12-16T11:36:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:36:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_49 2023-12-16T11:36:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:36:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-12-16T11:36:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-12-16T11:36:49.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:49.429+0000 7fee8c4c9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55dc6374f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:49.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:36:49.429+0000 7fee934d7640 -1 rbd::mirror::ImageReplayer: 0x55dc627d4c80 [3/fc94afcb-ef75-4403-83fc-5aacc271b450] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_50 2023-12-16T11:36:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-12-16T11:36:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:36:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local snap=snap 2023-12-16T11:36:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-12-16T11:36:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing snap: 100% complete...done. 2023-12-16T11:36:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_50 2023-12-16T11:36:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:36:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:36:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:36:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_50 2023-12-16T11:36:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:36:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:36:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-12-16T11:36:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-12-16T11:36:51.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:51.424+0000 7f5cb833c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac211e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:36:51.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:36:51.424+0000 7f5cbeb49640 -1 rbd::mirror::ImageReplayer: 0x555f9a96b180 [3/0e617b2e-921c-479c-a203-4d86e513e03e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:36:52.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-16T11:36:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ testlog 'TEST: image deletions should propagate' 2023-12-16T11:36:52.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee -a /tmp/tmp.jvOAE3Umds/rbd-mirror.test.log 2023-12-16T11:36:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ date '+%F %T' 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo 2023-12-16 11:36:52 TEST: image deletions should propagate 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:2023-12-16 11:36:52 TEST: image deletions should propagate 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_count=0 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local count 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local last_count=0 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ true 2023-12-16T11:36:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 40 2023-12-16T11:36:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in `seq 1 40` 2023-12-16T11:36:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 1 -ne 1 2023-12-16T11:36:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-16T11:36:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep 'images: ' 2023-12-16T11:36:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cut '-d ' -f 2 2023-12-16T11:36:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ count=0 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 = 0 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state 2023-12-16T11:36:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 40 2023-12-16T11:36:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in `seq 1 40` 2023-12-16T11:36:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 1 -ne 1 2023-12-16T11:36:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-16T11:36:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep 'image health:' 2023-12-16T11:36:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cut '-d ' -f 3 2023-12-16T11:36:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: failed to query services: (13) Permission denied 2023-12-16T11:36:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ state=OK 2023-12-16T11:36:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test OK = ERROR 2023-12-16T11:36:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test OK = OK 2023-12-16T11:36:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 50 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_1 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_1 2023-12-16T11:36:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_1 2023-12-16T11:36:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-16T11:36:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-16T11:36:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-16T11:36:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:36:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_1 2023-12-16T11:36:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_1 2023-12-16T11:36:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-16T11:36:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-16T11:36:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_2 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_2 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_2 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-16T11:36:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-16T11:36:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_2 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_2 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-16T11:36:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-16T11:36:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_3 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:36:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:36:52.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:36:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:36:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:36:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:36:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_4 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_4 2023-12-16T11:36:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_4 2023-12-16T11:36:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-16T11:36:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-16T11:36:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_4 2023-12-16T11:36:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_4 2023-12-16T11:36:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-16T11:36:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-16T11:36:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_5 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_5 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_5 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-16T11:36:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-16T11:36:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:53.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-16T11:36:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_5 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_5 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-16T11:36:53.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-16T11:36:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_6 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_6 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_6 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-16T11:36:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-16T11:36:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_6 2023-12-16T11:36:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_6 2023-12-16T11:36:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-16T11:36:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_7 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_7 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_7 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-16T11:36:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-16T11:36:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-12-16T11:36:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_7 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_7 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-16T11:36:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-16T11:36:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_8 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_8 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_8 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-16T11:36:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-16T11:36:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-12-16T11:36:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_8 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_8 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-16T11:36:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-16T11:36:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_9 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_9 2023-12-16T11:36:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_9 2023-12-16T11:36:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-16T11:36:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-16T11:36:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_9 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_9 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-16T11:36:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-16T11:36:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_10 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_10 2023-12-16T11:36:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_10 2023-12-16T11:36:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-16T11:36:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-16T11:36:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_10 2023-12-16T11:36:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_10 2023-12-16T11:36:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-16T11:36:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-16T11:36:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_11 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_11 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_11 2023-12-16T11:36:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-16T11:36:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-16T11:36:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2023-12-16T11:36:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_11 2023-12-16T11:36:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_11 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_11 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-12-16T11:36:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-16T11:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_12 2023-12-16T11:36:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_12 2023-12-16T11:36:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_12 2023-12-16T11:36:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-16T11:36:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-16T11:36:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_12 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_12 2023-12-16T11:36:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_12 2023-12-16T11:36:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-12-16T11:36:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-16T11:36:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_13 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_13 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_13 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-16T11:36:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-16T11:36:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_13 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_13 2023-12-16T11:36:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_13 2023-12-16T11:36:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-12-16T11:36:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-16T11:36:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_14 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_14 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_14 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-16T11:36:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-16T11:36:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_14 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_14 2023-12-16T11:36:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_14 2023-12-16T11:36:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-12-16T11:36:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-16T11:36:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_15 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_15 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_15 2023-12-16T11:36:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-16T11:36:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-16T11:36:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_15 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_15 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_15 2023-12-16T11:36:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-12-16T11:36:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-16T11:36:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_16 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_16 2023-12-16T11:36:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_16 2023-12-16T11:36:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-16T11:36:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-16T11:36:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_16 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_16 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_16 2023-12-16T11:36:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-12-16T11:36:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-16T11:36:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_17 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_17 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_17 2023-12-16T11:36:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-16T11:36:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-16T11:36:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2023-12-16T11:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_17 2023-12-16T11:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_17 2023-12-16T11:36:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_17 2023-12-16T11:36:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-12-16T11:36:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_18 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_18 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_18 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-16T11:36:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-16T11:36:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_18 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_18 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_18 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-12-16T11:36:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_19 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_19 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_19 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-16T11:36:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-16T11:36:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_19 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_19 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_19 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-12-16T11:36:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_20 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:36:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_20 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_20 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-16T11:36:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-16T11:36:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_20 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_20 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_20 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-12-16T11:36:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_21 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_21 2023-12-16T11:36:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_21 2023-12-16T11:36:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-16T11:36:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-16T11:36:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2023-12-16T11:36:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_21 2023-12-16T11:36:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_21 2023-12-16T11:36:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_21 2023-12-16T11:36:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-12-16T11:36:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-16T11:36:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_22 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_22 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_22 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-16T11:36:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-16T11:36:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_22 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_22 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_22 2023-12-16T11:36:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-12-16T11:36:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-16T11:36:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_23 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_23 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_23 2023-12-16T11:36:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-16T11:36:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-16T11:36:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_23 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_23 2023-12-16T11:36:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_23 2023-12-16T11:36:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-12-16T11:36:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_24 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_24 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_24 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-16T11:36:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-16T11:36:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:56.918 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_24 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_24 2023-12-16T11:36:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_24 2023-12-16T11:36:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-12-16T11:36:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-16T11:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_25 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_25 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_25 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-16T11:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-16T11:36:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_25 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_25 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_25 2023-12-16T11:36:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-12-16T11:36:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-16T11:36:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_26 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_26 2023-12-16T11:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_26 2023-12-16T11:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-16T11:36:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-16T11:36:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_26 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_26 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_26 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-12-16T11:36:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_27 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_27 2023-12-16T11:36:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_27 2023-12-16T11:36:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-16T11:36:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-16T11:36:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_27 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_27 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_27 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-12-16T11:36:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-16T11:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_28 2023-12-16T11:36:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_28 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_28 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-16T11:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-16T11:36:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_28 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_28 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_28 2023-12-16T11:36:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-12-16T11:36:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-16T11:36:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_29 2023-12-16T11:36:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_29 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_29 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-16T11:36:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-16T11:36:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_29 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_29 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_29 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-12-16T11:36:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-16T11:36:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:58.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_30 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_30 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_30 2023-12-16T11:36:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-16T11:36:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-16T11:36:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:58.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_30 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_30 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_30 2023-12-16T11:36:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-12-16T11:36:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-16T11:36:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_31 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_31 2023-12-16T11:36:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_31 2023-12-16T11:36:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-16T11:36:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-16T11:36:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_31 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_31 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_31 2023-12-16T11:36:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-12-16T11:36:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-16T11:36:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_32 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_32 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_32 2023-12-16T11:36:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-16T11:36:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-16T11:36:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2023-12-16T11:36:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_32 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_32 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_32 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-12-16T11:36:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_33 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_33 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_33 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-16T11:36:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-16T11:36:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2023-12-16T11:36:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_33 2023-12-16T11:36:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_33 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_33 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-12-16T11:36:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_34 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:36:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_34 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_34 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-16T11:36:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-16T11:36:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_34 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_34 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_34 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-12-16T11:36:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-16T11:36:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_35 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_35 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_35 2023-12-16T11:36:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-16T11:36:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-16T11:36:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2023-12-16T11:36:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_35 2023-12-16T11:36:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_35 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_35 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-12-16T11:36:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_36 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_36 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_36 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-16T11:36:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-16T11:36:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2023-12-16T11:36:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_36 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_36 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_36 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-12-16T11:36:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_37 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_37 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_37 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-16T11:36:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-16T11:36:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_37 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_37 2023-12-16T11:36:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_37 2023-12-16T11:36:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-12-16T11:36:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-16T11:36:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_38 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_38 2023-12-16T11:36:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_38 2023-12-16T11:36:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-16T11:36:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-16T11:36:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2023-12-16T11:36:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_38 2023-12-16T11:36:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_38 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_38 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-12-16T11:36:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-16T11:36:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_39 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_39 2023-12-16T11:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_39 2023-12-16T11:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-16T11:36:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-16T11:36:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:36:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_39 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_39 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_39 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-12-16T11:36:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_40 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:36:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_40 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_40 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-16T11:36:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:36:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-16T11:36:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:36:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:36:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_40 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_40 2023-12-16T11:37:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_40 2023-12-16T11:37:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-12-16T11:37:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_41 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.035 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_41 2023-12-16T11:37:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_41 2023-12-16T11:37:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-16T11:37:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-16T11:37:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2023-12-16T11:37:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_41 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_41 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_41 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-12-16T11:37:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_42 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-12-16T11:37:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_42 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_42 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-16T11:37:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-16T11:37:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_42 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_42 2023-12-16T11:37:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_42 2023-12-16T11:37:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-12-16T11:37:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_43 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_43 2023-12-16T11:37:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_43 2023-12-16T11:37:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-16T11:37:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-16T11:37:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_43 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_43 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_43 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-12-16T11:37:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_44 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_44 2023-12-16T11:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_44 2023-12-16T11:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-16T11:37:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-16T11:37:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2023-12-16T11:37:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_44 2023-12-16T11:37:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_44 2023-12-16T11:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_44 2023-12-16T11:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-12-16T11:37:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_45 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_45 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_45 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-16T11:37:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-16T11:37:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_45 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_45 2023-12-16T11:37:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_45 2023-12-16T11:37:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-12-16T11:37:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-16T11:37:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_46 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_46 2023-12-16T11:37:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_46 2023-12-16T11:37:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-16T11:37:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-16T11:37:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_46 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_46 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_46 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-12-16T11:37:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-16T11:37:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_47 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_47 2023-12-16T11:37:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_47 2023-12-16T11:37:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-16T11:37:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-16T11:37:01.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:01.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:01.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_47 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_47 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_47 2023-12-16T11:37:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-12-16T11:37:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_48 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-12-16T11:37:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_48 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_48 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-16T11:37:01.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-16T11:37:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2023-12-16T11:37:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_48 2023-12-16T11:37:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_48 2023-12-16T11:37:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_48 2023-12-16T11:37:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-12-16T11:37:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_49 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:01.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_49 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_49 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-16T11:37:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-16T11:37:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2023-12-16T11:37:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_49 2023-12-16T11:37:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_49 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_49 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-12-16T11:37:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_50 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_50 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_50 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-16T11:37:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-16T11:37:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:01.652 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5060706396005068) > weight (0.25) 2023-12-16T11:37:01.652 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-16T11:37:01.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-16T11:37:01.653 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-16T11:37:01.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:37:01.649+0000 7f28ab4e6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 107882) UID: 1000 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_50 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_50 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_50 2023-12-16T11:37:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-12-16T11:37:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-16T11:37:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ testlog 'TEST: delete images during bootstrap' 2023-12-16T11:37:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee -a /tmp/tmp.jvOAE3Umds/rbd-mirror.test.log 2023-12-16T11:37:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ date '+%F %T' 2023-12-16T11:37:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo 2023-12-16 11:37:01 TEST: delete images during bootstrap 2023-12-16T11:37:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:2023-12-16 11:37:01 TEST: delete images during bootstrap 2023-12-16T11:37:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-12-16T11:37:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=image 2023-12-16T11:37:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-12-16T11:37:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd: mirroring is already configured for image mode 2023-12-16T11:37:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-12-16T11:37:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=image 2023-12-16T11:37:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-12-16T11:37:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd: mirroring is already configured for image mode 2023-12-16T11:37:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ start_mirror cluster1 2023-12-16T11:37:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance 2023-12-16T11:37:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set_cluster_instance cluster1 cluster instance 2023-12-16T11:37:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local val=cluster1 2023-12-16T11:37:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster_var_name=cluster 2023-12-16T11:37:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local instance_var_name=instance 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cluster=cluster1 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=cluster1 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' cluster1 = cluster1 ']' 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ instance=0 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval cluster=cluster1 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ cluster=cluster1 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ eval instance=0 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ instance=0 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=test 2023-12-16T11:37:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 1 10 2023-12-16T11:37:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:37:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_1 2023-12-16T11:37:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_1 512M 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_1 2023-12-16T11:37:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_1 2023-12-16T11:37:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:37:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-16T11:37:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:37:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-16T11:37:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_1': 2023-12-16T11:37:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 187622a468ec 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.187622a468ec 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:37:01 2023 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:37:01 2023 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:37:01 2023 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:37:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 9dcbbba8-cd70-4080-9e3c-edfe3bde851f 2023-12-16T11:37:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:37:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-16T11:37:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:37:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=10 2023-12-16T11:37:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:37:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:37:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 76893184 bytes 2023-12-16T11:37:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 102217728 bytes 2023-12-16T11:37:02.654 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-16T11:37:02.654 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:37:02.654 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-16T11:37:02.654 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-16T11:37:02.654 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-16T11:37:02.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-16T11:37:02.658 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-16T11:37:02.778 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 129800192 bytes 2023-12-16T11:37:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 164898816 bytes 2023-12-16T11:37:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 216731648 bytes 2023-12-16T11:37:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 258199552 bytes 2023-12-16T11:37:03.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 311591936 bytes 2023-12-16T11:37:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 382257152 bytes 2023-12-16T11:37:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 387795968 bytes 2023-12-16T11:37:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 426096640 bytes 2023-12-16T11:37:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 451745792 bytes 2023-12-16T11:37:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 468134912 bytes 2023-12-16T11:37:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 497396736 bytes 2023-12-16T11:37:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 520222720 bytes 2023-12-16T11:37:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 596115456 bytes 2023-12-16T11:37:04.985 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 622097408 bytes 2023-12-16T11:37:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 648006656 bytes 2023-12-16T11:37:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 652766208 bytes 2023-12-16T11:37:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 692167680 bytes 2023-12-16T11:37:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 727833600 bytes 2023-12-16T11:37:05.673 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7880819700780468) > weight (0.25) 2023-12-16T11:37:05.673 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4388373421498998) > weight (0.25) 2023-12-16T11:37:05.673 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-16T11:37:05.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-16T11:37:05.674 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-16T11:37:05.674 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:37:05.670+0000 7fe897e5c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 96538) UID: 1000 2023-12-16T11:37:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 764651520 bytes 2023-12-16T11:37:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 810722304 bytes 2023-12-16T11:37:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 850009088 bytes 2023-12-16T11:37:06.252 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 897564672 bytes 2023-12-16T11:37:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 953042944 bytes 2023-12-16T11:37:06.583 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 972191744 bytes 2023-12-16T11:37:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1009458176 bytes 2023-12-16T11:37:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1070241792 bytes 2023-12-16T11:37:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1153435648 bytes 2023-12-16T11:37:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1182982144 bytes 2023-12-16T11:37:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1216606208 bytes 2023-12-16T11:37:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1236225024 bytes 2023-12-16T11:37:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1252975616 bytes 2023-12-16T11:37:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1322116096 bytes 2023-12-16T11:37:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1345923072 bytes 2023-12-16T11:37:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1381775360 bytes 2023-12-16T11:37:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1455283200 bytes 2023-12-16T11:37:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1472943104 bytes 2023-12-16T11:37:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1593595904 bytes 2023-12-16T11:37:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1622578176 bytes 2023-12-16T11:37:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1639155712 bytes 2023-12-16T11:37:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1732733952 bytes 2023-12-16T11:37:09.674 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-16T11:37:09.675 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:37:09.675 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-16T11:37:09.675 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-16T11:37:09.675 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-16T11:37:09.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-16T11:37:09.681 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-12-16T11:37:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1840851968 bytes 2023-12-16T11:37:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1869774848 bytes 2023-12-16T11:37:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1914258432 bytes 2023-12-16T11:37:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1946667008 bytes 2023-12-16T11:37:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2047959040 bytes 2023-12-16T11:37:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2081232896 bytes 2023-12-16T11:37:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2103014400 bytes 2023-12-16T11:37:11.165 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2137334784 bytes 2023-12-16T11:37:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2208009216 bytes 2023-12-16T11:37:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2267588608 bytes 2023-12-16T11:37:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2288377856 bytes 2023-12-16T11:37:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2363311104 bytes 2023-12-16T11:37:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2412316672 bytes 2023-12-16T11:37:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:37:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_1 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_1 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-16T11:37:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=10c53a1810c9 2023-12-16T11:37:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_1 present 10c53a1810c9 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=10c53a1810c9 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_1 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_1 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-16T11:37:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=10c53a1810c9 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 10c53a1810c9 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 10c53a1810c9 -o 10c53a1810c9 = 10c53a1810c9 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_1 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-12-16T11:37:13.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:37:13.182+0000 7ff4bdcce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x558f357a9000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-12-16T11:37:13.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:37:13.182+0000 7ff4bdcce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x558f357a9000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=367, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-12-16T11:37:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:37:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-16T11:37:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:37:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:37:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_1 2023-12-16T11:37:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_1 2023-12-16T11:37:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-16T11:37:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-16T11:37:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:37:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:37:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:37:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-12-16T11:37:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:37:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:37:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:37:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:37:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:37:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_1 2023-12-16T11:37:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:37:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:37:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_1 2023-12-16T11:37:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-12-16T11:37:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:37:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-16T11:37:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:37:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:37:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:37:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_1 2023-12-16T11:37:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-12-16T11:37:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-16T11:37:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:37:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:37:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_1 2023-12-16T11:37:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-16T11:37:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-16T11:37:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:37:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:37:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:37:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:37:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:37:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_1 2023-12-16T11:37:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-16T11:37:19.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-16T11:37:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:37:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:37:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:37:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:37:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:37:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_1 2023-12-16T11:37:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-16T11:37:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-16T11:37:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:37:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:37:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:37:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:37:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:37:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_1 2023-12-16T11:37:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-16T11:37:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-16T11:37:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:37:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:37:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:37:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:37:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:37:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_1 2023-12-16T11:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-16T11:37:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-16T11:37:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:37:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:37:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:37:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:37:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:37:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_1 2023-12-16T11:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_1 2023-12-16T11:37:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-16T11:37:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-16T11:37:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_2 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_2 512M 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:37:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_2 2023-12-16T11:37:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_2 2023-12-16T11:37:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:37:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:37:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-16T11:37:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:37:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-16T11:37:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_2': 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 18bc619b0cd3 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.18bc619b0cd3 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:37:50 2023 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:37:50 2023 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:37:50 2023 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 3f957c94-661a-470f-a338-811cbd98c031 2023-12-16T11:37:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:37:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-16T11:37:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:37:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:37:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:37:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:37:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:37:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:37:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:37:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 69561344 bytes 2023-12-16T11:37:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 126743552 bytes 2023-12-16T11:37:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 192489472 bytes 2023-12-16T11:37:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 241452032 bytes 2023-12-16T11:37:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 259361792 bytes 2023-12-16T11:37:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 348088320 bytes 2023-12-16T11:37:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 380033024 bytes 2023-12-16T11:37:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 398959616 bytes 2023-12-16T11:37:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 449120256 bytes 2023-12-16T11:37:53.102 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 491049984 bytes 2023-12-16T11:37:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 569922560 bytes 2023-12-16T11:37:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 655834112 bytes 2023-12-16T11:37:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 696569856 bytes 2023-12-16T11:37:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 739625984 bytes 2023-12-16T11:37:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 780342272 bytes 2023-12-16T11:37:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 881099776 bytes 2023-12-16T11:37:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 901741568 bytes 2023-12-16T11:37:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 922615808 bytes 2023-12-16T11:37:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 950113280 bytes 2023-12-16T11:37:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1011288064 bytes 2023-12-16T11:37:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1092492288 bytes 2023-12-16T11:37:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1118257152 bytes 2023-12-16T11:37:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1138196480 bytes 2023-12-16T11:37:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1160044544 bytes 2023-12-16T11:37:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1176773632 bytes 2023-12-16T11:37:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1191407616 bytes 2023-12-16T11:37:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1242578944 bytes 2023-12-16T11:37:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1280321536 bytes 2023-12-16T11:37:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1385175040 bytes 2023-12-16T11:37:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1452086272 bytes 2023-12-16T11:37:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1519212544 bytes 2023-12-16T11:37:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1576024064 bytes 2023-12-16T11:37:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1619049472 bytes 2023-12-16T11:37:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1633420288 bytes 2023-12-16T11:37:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1659088896 bytes 2023-12-16T11:37:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1687964672 bytes 2023-12-16T11:37:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1762580480 bytes 2023-12-16T11:37:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1788644352 bytes 2023-12-16T11:37:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1836684288 bytes 2023-12-16T11:37:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1907486720 bytes 2023-12-16T11:37:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1943860224 bytes 2023-12-16T11:37:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1971901440 bytes 2023-12-16T11:37:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 2063541248 bytes 2023-12-16T11:37:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2094989312 bytes 2023-12-16T11:37:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2131446784 bytes 2023-12-16T11:37:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2215230464 bytes 2023-12-16T11:37:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2238203904 bytes 2023-12-16T11:38:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2270792704 bytes 2023-12-16T11:38:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2310992896 bytes 2023-12-16T11:38:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2376945664 bytes 2023-12-16T11:38:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2395179008 bytes 2023-12-16T11:38:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2432922624 bytes 2023-12-16T11:38:00.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2508533760 bytes 2023-12-16T11:38:01.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2543836160 bytes 2023-12-16T11:38:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2592754688 bytes 2023-12-16T11:38:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2639108096 bytes 2023-12-16T11:38:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2673417216 bytes 2023-12-16T11:38:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2692220928 bytes 2023-12-16T11:38:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2713926656 bytes 2023-12-16T11:38:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2738095104 bytes 2023-12-16T11:38:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2783856640 bytes 2023-12-16T11:38:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2829624320 bytes 2023-12-16T11:38:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2848854016 bytes 2023-12-16T11:38:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2909680640 bytes 2023-12-16T11:38:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2911562752 bytes 2023-12-16T11:38:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2928300032 bytes 2023-12-16T11:38:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2967590912 bytes 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_2 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_2 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-16T11:38:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=10c5aa0d88cc 2023-12-16T11:38:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:38:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:38:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_2 present 10c5aa0d88cc 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=10c5aa0d88cc 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_2 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_2 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-16T11:38:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=10c5aa0d88cc 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 10c5aa0d88cc 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 10c5aa0d88cc -o 10c5aa0d88cc = 10c5aa0d88cc 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_2 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:03.433 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-12-16T11:38:04.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:38:04.224+0000 7ff4bdcce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x558f352d4000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-12-16T11:38:04.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:38:04.224+0000 7ff4bdcce640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x558f352d4000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-12-16T11:38:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:38:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-16T11:38:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:38:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:38:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:38:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:38:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_2 2023-12-16T11:38:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_2 2023-12-16T11:38:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-16T11:38:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-16T11:38:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:38:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:38:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:38:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-12-16T11:38:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:38:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:38:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:38:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:38:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_2 2023-12-16T11:38:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_2 2023-12-16T11:38:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-12-16T11:38:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-16T11:38:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:38:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:38:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:38:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:38:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_2 2023-12-16T11:38:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:09.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-12-16T11:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-16T11:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:38:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_2 2023-12-16T11:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-16T11:38:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-16T11:38:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:38:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:38:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:38:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:38:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:38:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_2 2023-12-16T11:38:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-16T11:38:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-16T11:38:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:38:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:38:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:38:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:38:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:38:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_2 2023-12-16T11:38:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-16T11:38:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-16T11:38:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:38:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:38:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:38:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:38:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_2 2023-12-16T11:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-16T11:38:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-16T11:38:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:38:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:38:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:38:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:38:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:38:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_2 2023-12-16T11:38:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-16T11:38:25.112 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-16T11:38:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:38:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:38:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:38:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:38:31.681 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.40511763959119707) > weight (0.25) 2023-12-16T11:38:31.681 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.436492398668585) > weight (0.25) 2023-12-16T11:38:31.681 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4705056996266661) > weight (0.25) 2023-12-16T11:38:31.682 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-16T11:38:31.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-16T11:38:31.682 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-16T11:38:31.682 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-16T11:38:31.682 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:38:31.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:38:31.679+0000 7ff4c9ce6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 91589) UID: 1000 2023-12-16T11:38:37.784 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-16T11:38:37.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-16T11:38:37.785 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-16T11:38:37.788 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-16T11:38:37.788 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-12-16T11:38:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_2 2023-12-16T11:38:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_2 2023-12-16T11:38:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-16T11:38:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-16T11:38:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_3 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_3 512M 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:38:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:38:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:38:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_3 2023-12-16T11:38:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_3 2023-12-16T11:38:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:38:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:38:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-16T11:38:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:38:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_3': 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 18f7a8161b55 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.18f7a8161b55 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:38:41 2023 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:38:41 2023 2023-12-16T11:38:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:38:41 2023 2023-12-16T11:38:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:38:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:38:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 60949d8e-5a6d-434a-9976-938bda2f407e 2023-12-16T11:38:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:38:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-16T11:38:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:38:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:38:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:38:42.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:38:42.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:38:42.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:38:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 54088704 bytes 2023-12-16T11:38:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 76997632 bytes 2023-12-16T11:38:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 94342144 bytes 2023-12-16T11:38:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 127712256 bytes 2023-12-16T11:38:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 153117696 bytes 2023-12-16T11:38:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 210190336 bytes 2023-12-16T11:38:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 239063040 bytes 2023-12-16T11:38:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 291990528 bytes 2023-12-16T11:38:43.738 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 345793536 bytes 2023-12-16T11:38:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 385093632 bytes 2023-12-16T11:38:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 403722240 bytes 2023-12-16T11:38:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 445445120 bytes 2023-12-16T11:38:44.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 514674688 bytes 2023-12-16T11:38:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 544186368 bytes 2023-12-16T11:38:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 574375936 bytes 2023-12-16T11:38:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 641163264 bytes 2023-12-16T11:38:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 667463680 bytes 2023-12-16T11:38:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 731974656 bytes 2023-12-16T11:38:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 793177088 bytes 2023-12-16T11:38:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 825647104 bytes 2023-12-16T11:38:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 833035264 bytes 2023-12-16T11:38:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 860104704 bytes 2023-12-16T11:38:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 878801920 bytes 2023-12-16T11:38:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 917002240 bytes 2023-12-16T11:38:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 960865280 bytes 2023-12-16T11:38:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 998069248 bytes 2023-12-16T11:38:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1015549952 bytes 2023-12-16T11:38:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1069630464 bytes 2023-12-16T11:38:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1100937216 bytes 2023-12-16T11:38:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1106810880 bytes 2023-12-16T11:38:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1150069760 bytes 2023-12-16T11:38:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1205553152 bytes 2023-12-16T11:38:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1218255872 bytes 2023-12-16T11:38:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1223933952 bytes 2023-12-16T11:38:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1266655232 bytes 2023-12-16T11:38:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1289111552 bytes 2023-12-16T11:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1325641728 bytes 2023-12-16T11:38:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1358867456 bytes 2023-12-16T11:38:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1381072896 bytes 2023-12-16T11:38:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1396608000 bytes 2023-12-16T11:38:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1415642112 bytes 2023-12-16T11:38:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1451949056 bytes 2023-12-16T11:38:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1498768384 bytes 2023-12-16T11:38:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1565299712 bytes 2023-12-16T11:38:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1609564160 bytes 2023-12-16T11:38:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1642022912 bytes 2023-12-16T11:38:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1722874880 bytes 2023-12-16T11:38:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1738925056 bytes 2023-12-16T11:38:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1833928704 bytes 2023-12-16T11:38:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1860023296 bytes 2023-12-16T11:38:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1897001984 bytes 2023-12-16T11:38:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 1931398144 bytes 2023-12-16T11:38:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 1977784320 bytes 2023-12-16T11:38:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2015936512 bytes 2023-12-16T11:38:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2061477888 bytes 2023-12-16T11:38:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2147909632 bytes 2023-12-16T11:38:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2156060672 bytes 2023-12-16T11:38:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2205072384 bytes 2023-12-16T11:38:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2219403264 bytes 2023-12-16T11:38:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2286722048 bytes 2023-12-16T11:38:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2353088512 bytes 2023-12-16T11:38:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2400364544 bytes 2023-12-16T11:38:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2402121728 bytes 2023-12-16T11:38:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2418707456 bytes 2023-12-16T11:38:53.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2448851968 bytes 2023-12-16T11:38:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2500406272 bytes 2023-12-16T11:38:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2561030144 bytes 2023-12-16T11:38:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2582979584 bytes 2023-12-16T11:38:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2608062464 bytes 2023-12-16T11:38:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2643722240 bytes 2023-12-16T11:38:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 2710659072 bytes 2023-12-16T11:38:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:38:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:38:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:38:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:38:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:38:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:38:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_3 present '' 2023-12-16T11:38:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:38:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:38:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = deleted 2023-12-16T11:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:38:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_3 present '' 2023-12-16T11:38:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:38:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:38:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:38:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:38:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = deleted 2023-12-16T11:38:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:38:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_3 present '' 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:38:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:38:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:38:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:38:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:38:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:38:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:38:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:38:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:38:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = deleted 2023-12-16T11:38:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:38:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:39:00.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29609054155040637) > weight (0.25) 2023-12-16T11:39:00.659 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.48498098232604603) > weight (0.25) 2023-12-16T11:39:00.659 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2990605399218317) > weight (0.25) 2023-12-16T11:39:00.659 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-16T11:39:00.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-16T11:39:00.659 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-16T11:39:00.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:39:00.657+0000 7fee93cd8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 103754) UID: 1000 2023-12-16T11:39:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_3 present '' 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:39:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:39:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:39:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:39:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:39:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = deleted 2023-12-16T11:39:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:39:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:39:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_3 present '' 2023-12-16T11:39:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:39:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c84737ccd3 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c84737ccd3 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z '' -o '' = 12c84737ccd3 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_3 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-12-16T11:39:09.660 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-16T11:39:09.661 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:39:09.661 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-16T11:39:09.661 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-16T11:39:09.661 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-16T11:39:09.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-16T11:39:09.664 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-16T11:39:10.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:39:10.114+0000 7f5eae982640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fead0ef800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-12-16T11:39:10.118 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:39:10.114+0000 7f5eae982640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fead0ef800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-12-16T11:39:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:39:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-16T11:39:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:39:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:39:15.169 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:39:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:39:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:39:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_3 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_3 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-12-16T11:39:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_3 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-12-16T11:39:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-16T11:39:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:39:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_3 2023-12-16T11:39:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-16T11:39:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-16T11:39:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:39:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_3 2023-12-16T11:39:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-16T11:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-16T11:39:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:39:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_3 2023-12-16T11:39:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-16T11:39:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-16T11:39:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:39:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_3 2023-12-16T11:39:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-16T11:39:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-16T11:39:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:39:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_3 2023-12-16T11:39:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_3 2023-12-16T11:39:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-16T11:39:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-16T11:39:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:39:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:39:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:39:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_4 2023-12-16T11:39:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_4 512M 2023-12-16T11:39:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:39:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:39:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_4 2023-12-16T11:39:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_4 2023-12-16T11:39:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:39:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-16T11:39:32.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:39:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-16T11:39:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_4': 2023-12-16T11:39:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:39:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:39:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 19319d0f0850 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.19319d0f0850 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:39:31 2023 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:39:31 2023 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:39:31 2023 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 63458725-aba0-4dac-afcd-db6b484da6d0 2023-12-16T11:39:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:39:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-16T11:39:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:39:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=7 2023-12-16T11:39:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:39:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:39:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 35775488 bytes 2023-12-16T11:39:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 75944960 bytes 2023-12-16T11:39:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 98248704 bytes 2023-12-16T11:39:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 122002432 bytes 2023-12-16T11:39:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 167714816 bytes 2023-12-16T11:39:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 173268992 bytes 2023-12-16T11:39:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 231406592 bytes 2023-12-16T11:39:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 255507456 bytes 2023-12-16T11:39:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 265802752 bytes 2023-12-16T11:39:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 343872512 bytes 2023-12-16T11:39:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 384746496 bytes 2023-12-16T11:39:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 419990528 bytes 2023-12-16T11:39:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 499054592 bytes 2023-12-16T11:39:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 519788544 bytes 2023-12-16T11:39:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 578398208 bytes 2023-12-16T11:39:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 647280640 bytes 2023-12-16T11:39:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 698217472 bytes 2023-12-16T11:39:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 736402432 bytes 2023-12-16T11:39:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 760639488 bytes 2023-12-16T11:39:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 804847616 bytes 2023-12-16T11:39:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 824890368 bytes 2023-12-16T11:39:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 888453120 bytes 2023-12-16T11:39:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 933637120 bytes 2023-12-16T11:39:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 969492480 bytes 2023-12-16T11:39:36.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1011580928 bytes 2023-12-16T11:39:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1042145280 bytes 2023-12-16T11:39:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1096589312 bytes 2023-12-16T11:39:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1153769472 bytes 2023-12-16T11:39:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1182073856 bytes 2023-12-16T11:39:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1241970688 bytes 2023-12-16T11:39:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1300790272 bytes 2023-12-16T11:39:37.843 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:39:37.839+0000 7f6f756e7640 -1 reset not still connected to 0x5577c35cbee0 2023-12-16T11:39:37.994 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1364948992 bytes 2023-12-16T11:39:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1375507456 bytes 2023-12-16T11:39:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1398056960 bytes 2023-12-16T11:39:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1411273728 bytes 2023-12-16T11:39:38.478 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1422712832 bytes 2023-12-16T11:39:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1530471424 bytes 2023-12-16T11:39:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1550577664 bytes 2023-12-16T11:39:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1574003712 bytes 2023-12-16T11:39:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1640483840 bytes 2023-12-16T11:39:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:39:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:39:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:39:39.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_4 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_4 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-16T11:39:39.483 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=12c8826f4203 2023-12-16T11:39:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:39:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:39:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_4 present 12c8826f4203 2023-12-16T11:39:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=12c8826f4203 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_4 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_4 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-16T11:39:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c8826f4203 2023-12-16T11:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c8826f4203 2023-12-16T11:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 12c8826f4203 -o 12c8826f4203 = 12c8826f4203 2023-12-16T11:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:39:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_4 2023-12-16T11:39:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-12-16T11:39:40.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:39:40.163+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fea95cc400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-12-16T11:39:40.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:39:40.163+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fea95cc400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-12-16T11:39:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:39:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_4 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_4 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-16T11:39:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-16T11:39:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:39:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:39:49.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:39:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-12-16T11:39:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_4 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_4 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-12-16T11:39:49.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:39:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-16T11:39:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_4 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-12-16T11:39:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-16T11:39:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:39:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_4 2023-12-16T11:39:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-16T11:39:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-16T11:39:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:39:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_4 2023-12-16T11:39:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-16T11:39:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-16T11:39:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:39:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_4 2023-12-16T11:39:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-16T11:39:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-16T11:39:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:39:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_4 2023-12-16T11:39:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:39:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:39:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:39:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-16T11:39:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-16T11:39:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:39:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:39:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:39:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:39:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:39:59.788 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4199678797504468) > weight (0.25) 2023-12-16T11:39:59.789 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.733384565323323) > weight (0.25) 2023-12-16T11:39:59.789 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6912534068685221) > weight (0.25) 2023-12-16T11:39:59.789 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6433425653058621) > weight (0.25) 2023-12-16T11:39:59.789 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2023-12-16T11:40:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_4 2023-12-16T11:40:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:40:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-16T11:40:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-16T11:40:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:40:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:40:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:40:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:40:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_4 2023-12-16T11:40:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_4 2023-12-16T11:40:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-16T11:40:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-16T11:40:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_5 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_5 512M 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:40:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:40:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:40:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:40:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:40:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:40:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_5 2023-12-16T11:40:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_5 2023-12-16T11:40:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:40:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-16T11:40:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:40:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_5': 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 19672bded4f6 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.19672bded4f6 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:40:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:40:21 2023 2023-12-16T11:40:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:40:21 2023 2023-12-16T11:40:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:40:21 2023 2023-12-16T11:40:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:40:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:40:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 7ce397c4-011c-4a70-a55b-58a260e7569e 2023-12-16T11:40:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:40:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-16T11:40:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:40:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=5 2023-12-16T11:40:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:40:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:40:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 628736 bytes 2023-12-16T11:40:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 24332288 bytes 2023-12-16T11:40:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 41605120 bytes 2023-12-16T11:40:22.664 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.580720439663273) > weight (0.25) 2023-12-16T11:40:22.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4355812458564331) > weight (0.25) 2023-12-16T11:40:22.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.540814556472243) > weight (0.25) 2023-12-16T11:40:22.665 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.34246224563210104) > weight (0.25) 2023-12-16T11:40:22.665 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-16T11:40:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 116302848 bytes 2023-12-16T11:40:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 158014464 bytes 2023-12-16T11:40:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 190801920 bytes 2023-12-16T11:40:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 191438848 bytes 2023-12-16T11:40:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 215472128 bytes 2023-12-16T11:40:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 264974336 bytes 2023-12-16T11:40:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 322285568 bytes 2023-12-16T11:40:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 398656512 bytes 2023-12-16T11:40:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 431887360 bytes 2023-12-16T11:40:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 474209280 bytes 2023-12-16T11:40:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 566671360 bytes 2023-12-16T11:40:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 597662720 bytes 2023-12-16T11:40:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 635199488 bytes 2023-12-16T11:40:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 696126464 bytes 2023-12-16T11:40:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 718756864 bytes 2023-12-16T11:40:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 786794496 bytes 2023-12-16T11:40:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 855537664 bytes 2023-12-16T11:40:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 879093760 bytes 2023-12-16T11:40:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 903785472 bytes 2023-12-16T11:40:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 943323136 bytes 2023-12-16T11:40:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1001484288 bytes 2023-12-16T11:40:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1018356736 bytes 2023-12-16T11:40:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1052369920 bytes 2023-12-16T11:40:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1123225600 bytes 2023-12-16T11:40:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1143816192 bytes 2023-12-16T11:40:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1173217280 bytes 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_5 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_5 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-16T11:40:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:40:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=12c8dbca06f0 2023-12-16T11:40:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:40:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_5 present 12c8dbca06f0 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=12c8dbca06f0 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_5 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:40:27.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:40:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_5 2023-12-16T11:40:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-16T11:40:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:40:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c8dbca06f0 2023-12-16T11:40:27.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c8dbca06f0 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 12c8dbca06f0 -o 12c8dbca06f0 = 12c8dbca06f0 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_5 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-12-16T11:40:28.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:40:28.284+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa5ee400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-12-16T11:40:28.288 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:40:28.284+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa5ee400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-12-16T11:40:32.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:40:32.752+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354ec14ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:40:32.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:40:32.752+0000 7f1c826b7640 -1 rbd::mirror::ImageReplayer: 0x56354eec4280 [3/7ce397c4-011c-4a70-a55b-58a260e7569e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:40:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:40:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-16T11:40:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:40:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:40:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:40:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:40:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:40:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_5 2023-12-16T11:40:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:40:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:40:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_5 2023-12-16T11:40:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-16T11:40:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:40:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-16T11:40:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:40:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:40:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_5 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:40:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_5 2023-12-16T11:40:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-12-16T11:40:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:40:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_5 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-12-16T11:40:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-16T11:40:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:40:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:40:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_5 2023-12-16T11:40:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-16T11:40:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-16T11:40:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:40:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:40:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:40:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:40:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:40:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_5 2023-12-16T11:40:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-16T11:40:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-16T11:40:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:40:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:40:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:40:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:40:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:40:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_5 2023-12-16T11:40:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-16T11:40:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-16T11:40:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:40:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:40:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:40:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:40:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_5 2023-12-16T11:40:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-16T11:40:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-16T11:40:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:40:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:40:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:40:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:40:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:40:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_5 2023-12-16T11:40:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:40:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:40:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:40:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-16T11:40:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-16T11:40:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:40:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:40:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:40:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:40:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:41:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_5 2023-12-16T11:41:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_5 2023-12-16T11:41:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-16T11:41:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-16T11:41:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_6 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_6 512M 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_6 2023-12-16T11:41:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_6 2023-12-16T11:41:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:41:05.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-16T11:41:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:41:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-16T11:41:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_6': 2023-12-16T11:41:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:41:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:41:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:41:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 199d3391a4ce 2023-12-16T11:41:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.199d3391a4ce 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:41:05 2023 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:41:05 2023 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:41:05 2023 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: b042891f-3c04-45e3-90fd-c5ba56a424a2 2023-12-16T11:41:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:41:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-16T11:41:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:06.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:41:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=8 2023-12-16T11:41:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:41:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:41:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 62907392 bytes 2023-12-16T11:41:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 81021952 bytes 2023-12-16T11:41:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 95611904 bytes 2023-12-16T11:41:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 145065984 bytes 2023-12-16T11:41:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 185874432 bytes 2023-12-16T11:41:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 208023552 bytes 2023-12-16T11:41:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 239944704 bytes 2023-12-16T11:41:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 296934400 bytes 2023-12-16T11:41:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 354148352 bytes 2023-12-16T11:41:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 374040576 bytes 2023-12-16T11:41:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 381626368 bytes 2023-12-16T11:41:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 390451200 bytes 2023-12-16T11:41:08.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 501619712 bytes 2023-12-16T11:41:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 530988032 bytes 2023-12-16T11:41:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 580212736 bytes 2023-12-16T11:41:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 619638784 bytes 2023-12-16T11:41:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 697574400 bytes 2023-12-16T11:41:09.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 728508416 bytes 2023-12-16T11:41:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 787117056 bytes 2023-12-16T11:41:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 818780160 bytes 2023-12-16T11:41:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 854024192 bytes 2023-12-16T11:41:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 871078912 bytes 2023-12-16T11:41:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 902073344 bytes 2023-12-16T11:41:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 973408256 bytes 2023-12-16T11:41:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1103541248 bytes 2023-12-16T11:41:10.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1124801536 bytes 2023-12-16T11:41:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1163844608 bytes 2023-12-16T11:41:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1195343872 bytes 2023-12-16T11:41:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1246779392 bytes 2023-12-16T11:41:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1253012480 bytes 2023-12-16T11:41:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1333230592 bytes 2023-12-16T11:41:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1362494464 bytes 2023-12-16T11:41:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1417277440 bytes 2023-12-16T11:41:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1510994944 bytes 2023-12-16T11:41:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1569555456 bytes 2023-12-16T11:41:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1619634176 bytes 2023-12-16T11:41:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1713786880 bytes 2023-12-16T11:41:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1788624896 bytes 2023-12-16T11:41:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1811622912 bytes 2023-12-16T11:41:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1856877568 bytes 2023-12-16T11:41:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1900887040 bytes 2023-12-16T11:41:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1911430144 bytes 2023-12-16T11:41:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1913937920 bytes 2023-12-16T11:41:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1991534592 bytes 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_6 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_6 2023-12-16T11:41:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-16T11:41:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:41:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=12c88850d95f 2023-12-16T11:41:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:41:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_6 present 12c88850d95f 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=12c88850d95f 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_6 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_6 2023-12-16T11:41:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-16T11:41:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c88850d95f 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c88850d95f 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 12c88850d95f -o 12c88850d95f = 12c88850d95f 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_6 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:14.722 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-12-16T11:41:15.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:41:15.264+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fea9224c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-12-16T11:41:15.268 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:41:15.264+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55fea9224c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-12-16T11:41:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:41:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-16T11:41:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:41:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:41:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:41:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:41:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:41:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_6 2023-12-16T11:41:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:41:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:41:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_6 2023-12-16T11:41:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-16T11:41:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:41:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-16T11:41:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:41:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:41:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_6 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_6 2023-12-16T11:41:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-12-16T11:41:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:41:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-16T11:41:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_6 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-12-16T11:41:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-16T11:41:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:41:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:41:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_6 2023-12-16T11:41:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-16T11:41:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-16T11:41:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:41:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:41:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:41:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:41:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:41:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_6 2023-12-16T11:41:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-16T11:41:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-16T11:41:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:41:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:41:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:41:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:41:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:41:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_6 2023-12-16T11:41:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-16T11:41:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-16T11:41:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:41:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:41:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:41:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:41:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:41:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_6 2023-12-16T11:41:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-16T11:41:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-16T11:41:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:41:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:41:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:41:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:41:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_6 2023-12-16T11:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-16T11:41:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-16T11:41:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:41:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:41:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:41:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:41:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:41:45.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6209718063271251) > weight (0.25) 2023-12-16T11:41:45.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4859582991693454) > weight (0.25) 2023-12-16T11:41:45.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5964105113203053) > weight (0.25) 2023-12-16T11:41:45.666 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4914042920974191) > weight (0.25) 2023-12-16T11:41:45.666 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-16T11:41:48.789 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8034940812962016) > weight (0.25) 2023-12-16T11:41:48.790 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7331721591496942) > weight (0.25) 2023-12-16T11:41:48.790 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3607618655574828) > weight (0.25) 2023-12-16T11:41:48.790 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-16T11:41:48.790 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-16T11:41:48.791 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-16T11:41:48.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:41:48.789+0000 7fe77a127640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 125088) UID: 1000 2023-12-16T11:41:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_6 2023-12-16T11:41:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_6 2023-12-16T11:41:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-16T11:41:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-16T11:41:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_7 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_7 512M 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:41:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:41:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:41:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:41:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_7 2023-12-16T11:41:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_7 2023-12-16T11:41:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:41:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:41:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-16T11:41:52.791 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-16T11:41:52.791 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:41:52.792 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-16T11:41:52.792 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-16T11:41:52.792 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-16T11:41:52.795 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-16T11:41:52.795 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-16T11:41:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:41:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_7': 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 19d3a379cf8a 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.19d3a379cf8a 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:41:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:41:52 2023 2023-12-16T11:41:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:41:52 2023 2023-12-16T11:41:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:41:52 2023 2023-12-16T11:41:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:41:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:41:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: e346edb6-bd21-4358-894a-4569902b2213 2023-12-16T11:41:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:41:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-16T11:41:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:41:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:41:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:41:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:41:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=8 2023-12-16T11:41:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:41:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:41:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 32995328 bytes 2023-12-16T11:41:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 89841664 bytes 2023-12-16T11:41:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 134868992 bytes 2023-12-16T11:41:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 196499456 bytes 2023-12-16T11:41:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 255312896 bytes 2023-12-16T11:41:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 286009344 bytes 2023-12-16T11:41:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 368992256 bytes 2023-12-16T11:41:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 375232512 bytes 2023-12-16T11:41:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 406964224 bytes 2023-12-16T11:41:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 410314752 bytes 2023-12-16T11:41:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 480142336 bytes 2023-12-16T11:41:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 507629568 bytes 2023-12-16T11:41:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 554173440 bytes 2023-12-16T11:41:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 642071552 bytes 2023-12-16T11:41:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 646456320 bytes 2023-12-16T11:41:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 685980672 bytes 2023-12-16T11:41:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 716388352 bytes 2023-12-16T11:41:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 756750336 bytes 2023-12-16T11:41:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 823981056 bytes 2023-12-16T11:41:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 843488256 bytes 2023-12-16T11:41:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 888448000 bytes 2023-12-16T11:41:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 909592576 bytes 2023-12-16T11:41:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 958024704 bytes 2023-12-16T11:41:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1012222976 bytes 2023-12-16T11:41:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1041243136 bytes 2023-12-16T11:41:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1126111232 bytes 2023-12-16T11:41:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1155227648 bytes 2023-12-16T11:41:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1206875136 bytes 2023-12-16T11:41:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1228036096 bytes 2023-12-16T11:41:58.595 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1275801600 bytes 2023-12-16T11:41:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1344668672 bytes 2023-12-16T11:41:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1345302528 bytes 2023-12-16T11:41:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1372046336 bytes 2023-12-16T11:41:59.208 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1403154432 bytes 2023-12-16T11:41:59.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1423282176 bytes 2023-12-16T11:41:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1443115008 bytes 2023-12-16T11:41:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1504301056 bytes 2023-12-16T11:41:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1550681088 bytes 2023-12-16T11:41:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1558012928 bytes 2023-12-16T11:42:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1627134976 bytes 2023-12-16T11:42:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1654597632 bytes 2023-12-16T11:42:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1678341120 bytes 2023-12-16T11:42:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1684738048 bytes 2023-12-16T11:42:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1727950848 bytes 2023-12-16T11:42:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1776663552 bytes 2023-12-16T11:42:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1827367936 bytes 2023-12-16T11:42:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1829189632 bytes 2023-12-16T11:42:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:42:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:42:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:42:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:42:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:42:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_7 2023-12-16T11:42:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_7 2023-12-16T11:42:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-16T11:42:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=12c8dab2c108 2023-12-16T11:42:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:42:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_7 present 12c8dab2c108 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=12c8dab2c108 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_7 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_7 2023-12-16T11:42:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-16T11:42:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c8dab2c108 2023-12-16T11:42:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c8dab2c108 2023-12-16T11:42:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 12c8dab2c108 -o 12c8dab2c108 = 12c8dab2c108 2023-12-16T11:42:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:42:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:42:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:42:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_7 2023-12-16T11:42:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:01.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-12-16T11:42:02.272 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:42:02.269+0000 7f5eae982640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa1b3400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-12-16T11:42:02.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:42:02.269+0000 7f5eae982640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa1b3400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-12-16T11:42:02.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:42:02.752+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354ec149c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:42:02.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:42:02.752+0000 7f1c826b7640 -1 rbd::mirror::ImageReplayer: 0x56354eec4280 [3/e346edb6-bd21-4358-894a-4569902b2213] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:42:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:42:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-16T11:42:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_7 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_7 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-16T11:42:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-16T11:42:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:42:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:42:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_7 2023-12-16T11:42:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_7 2023-12-16T11:42:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-12-16T11:42:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_7 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-12-16T11:42:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-16T11:42:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:42:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_7 2023-12-16T11:42:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-16T11:42:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:07.806 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:07.809 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:42:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_7 2023-12-16T11:42:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-16T11:42:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-16T11:42:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:42:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_7 2023-12-16T11:42:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-16T11:42:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-16T11:42:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:42:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_7 2023-12-16T11:42:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-16T11:42:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-16T11:42:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:42:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_7 2023-12-16T11:42:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-16T11:42:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-16T11:42:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:42:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_7 2023-12-16T11:42:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_7 2023-12-16T11:42:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-16T11:42:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-16T11:42:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:42:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:42:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:42:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_8 2023-12-16T11:42:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_8 512M 2023-12-16T11:42:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:42:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:42:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:42:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:42:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:42:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:42:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:42:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_8 2023-12-16T11:42:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_8 2023-12-16T11:42:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:42:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-16T11:42:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-16T11:42:40.228 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_8': 2023-12-16T11:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 19ff70262878 2023-12-16T11:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.19ff70262878 2023-12-16T11:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:42:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:42:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:42:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:42:39 2023 2023-12-16T11:42:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:42:39 2023 2023-12-16T11:42:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:42:39 2023 2023-12-16T11:42:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:42:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:42:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 17f10e80-ca82-48d5-bdda-1c75e3c65c4b 2023-12-16T11:42:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:42:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-16T11:42:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:42:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=9 2023-12-16T11:42:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:42:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:42:40.435 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 35668992 bytes 2023-12-16T11:42:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 64636928 bytes 2023-12-16T11:42:40.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 151243776 bytes 2023-12-16T11:42:41.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 203616256 bytes 2023-12-16T11:42:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 204223488 bytes 2023-12-16T11:42:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 284215296 bytes 2023-12-16T11:42:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 375955456 bytes 2023-12-16T11:42:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 447455232 bytes 2023-12-16T11:42:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 471595008 bytes 2023-12-16T11:42:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 477856768 bytes 2023-12-16T11:42:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 554519552 bytes 2023-12-16T11:42:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 579769344 bytes 2023-12-16T11:42:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 644487168 bytes 2023-12-16T11:42:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 728976384 bytes 2023-12-16T11:42:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 768621568 bytes 2023-12-16T11:42:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 793631744 bytes 2023-12-16T11:42:43.442 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 839667712 bytes 2023-12-16T11:42:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 882064384 bytes 2023-12-16T11:42:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 925233152 bytes 2023-12-16T11:42:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 1013407744 bytes 2023-12-16T11:42:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 1035795456 bytes 2023-12-16T11:42:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 1102326784 bytes 2023-12-16T11:42:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 1146265600 bytes 2023-12-16T11:42:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1190475776 bytes 2023-12-16T11:42:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1215340544 bytes 2023-12-16T11:42:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1239214080 bytes 2023-12-16T11:42:45.166 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1288632320 bytes 2023-12-16T11:42:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1289269248 bytes 2023-12-16T11:42:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1353735168 bytes 2023-12-16T11:42:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1446322176 bytes 2023-12-16T11:42:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1496838144 bytes 2023-12-16T11:42:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1522073600 bytes 2023-12-16T11:42:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1544165376 bytes 2023-12-16T11:42:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1569102848 bytes 2023-12-16T11:42:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1603593216 bytes 2023-12-16T11:42:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1654628352 bytes 2023-12-16T11:42:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1700992000 bytes 2023-12-16T11:42:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1794542592 bytes 2023-12-16T11:42:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1842120704 bytes 2023-12-16T11:42:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1881123840 bytes 2023-12-16T11:42:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1904519168 bytes 2023-12-16T11:42:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1905851392 bytes 2023-12-16T11:42:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1971710976 bytes 2023-12-16T11:42:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2032320512 bytes 2023-12-16T11:42:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2033923072 bytes 2023-12-16T11:42:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2050682880 bytes 2023-12-16T11:42:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2097781760 bytes 2023-12-16T11:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2111051776 bytes 2023-12-16T11:42:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2135822336 bytes 2023-12-16T11:42:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2178429952 bytes 2023-12-16T11:42:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:42:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_8 2023-12-16T11:42:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_8 2023-12-16T11:42:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-16T11:42:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=12c8b7bc4b15 2023-12-16T11:42:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:42:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_8 present 12c8b7bc4b15 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=12c8b7bc4b15 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_8 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_8 2023-12-16T11:42:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-16T11:42:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c8b7bc4b15 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c8b7bc4b15 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 12c8b7bc4b15 -o 12c8b7bc4b15 = 12c8b7bc4b15 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_8 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-12-16T11:42:50.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:42:50.187+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa19c400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-12-16T11:42:50.190 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:42:50.187+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa19c400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-12-16T11:42:53.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:42:53.820+0000 7f1c7beaa640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56354f23c340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:42:53.841 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:42:53.820+0000 7f1c826b7640 -1 rbd::mirror::ImageReplayer: 0x56354eec4280 [3/17f10e80-ca82-48d5-bdda-1c75e3c65c4b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:42:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:42:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_8 2023-12-16T11:42:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_8 2023-12-16T11:42:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-16T11:42:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-16T11:42:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:42:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:42:55.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_8 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:42:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_8 2023-12-16T11:42:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-12-16T11:42:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:42:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_8 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-12-16T11:42:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-16T11:42:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_8 2023-12-16T11:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-16T11:42:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-16T11:42:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:55.510 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:42:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_8 2023-12-16T11:42:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-16T11:42:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-16T11:42:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:42:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_8 2023-12-16T11:42:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:42:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:42:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:42:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-16T11:42:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-16T11:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:42:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:42:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_8 2023-12-16T11:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:43:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-16T11:43:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-16T11:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:43:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:43:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:43:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:43:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_8 2023-12-16T11:43:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:43:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-16T11:43:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-16T11:43:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:43:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:43:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:43:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:43:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:43:21.796 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.992127076720808) > weight (0.25) 2023-12-16T11:43:21.796 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4890567032932587) > weight (0.25) 2023-12-16T11:43:21.797 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4244609234317023) > weight (0.25) 2023-12-16T11:43:21.797 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-16T11:43:21.797 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-16T11:43:21.797 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-16T11:43:21.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:43:21.796+0000 7f6a8148d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 127325) UID: 1000 2023-12-16T11:43:25.667 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-16T11:43:25.668 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-16T11:43:25.668 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-16T11:43:25.668 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-16T11:43:25.668 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:43:25.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:43:25.666+0000 7f1c82eb8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 101868) UID: 1000 2023-12-16T11:43:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_8 2023-12-16T11:43:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_8 2023-12-16T11:43:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-16T11:43:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-16T11:43:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:43:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_9 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_9 512M 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:43:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_9 2023-12-16T11:43:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_9 2023-12-16T11:43:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:43:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-16T11:43:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:43:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-16T11:43:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_9': 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1a2a25ab8252 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1a2a25ab8252 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:43:27 2023 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:43:27 2023 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:43:27 2023 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:43:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:43:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 54115782-4cc3-4ad5-803a-ca9c409d9c36 2023-12-16T11:43:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:43:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-16T11:43:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:43:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=14 2023-12-16T11:43:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:43:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 54223872 bytes 2023-12-16T11:43:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 61162496 bytes 2023-12-16T11:43:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 94557184 bytes 2023-12-16T11:43:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 150382592 bytes 2023-12-16T11:43:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 190850048 bytes 2023-12-16T11:43:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 244209664 bytes 2023-12-16T11:43:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 347296768 bytes 2023-12-16T11:43:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 379037696 bytes 2023-12-16T11:43:29.798 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-16T11:43:29.798 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:43:29.798 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-16T11:43:29.798 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-16T11:43:29.798 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-16T11:43:29.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-16T11:43:29.801 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2023-12-16T11:43:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 456886272 bytes 2023-12-16T11:43:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 477935616 bytes 2023-12-16T11:43:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 573352960 bytes 2023-12-16T11:43:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 600283136 bytes 2023-12-16T11:43:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 600883200 bytes 2023-12-16T11:43:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 673321984 bytes 2023-12-16T11:43:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 688166912 bytes 2023-12-16T11:43:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 688764928 bytes 2023-12-16T11:43:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 774688768 bytes 2023-12-16T11:43:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 787209216 bytes 2023-12-16T11:43:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 855007232 bytes 2023-12-16T11:43:31.770 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-16T11:43:31.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-16T11:43:31.770 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-16T11:43:31.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-16T11:43:31.773 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-12-16T11:43:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 882938880 bytes 2023-12-16T11:43:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 915229696 bytes 2023-12-16T11:43:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 979523584 bytes 2023-12-16T11:43:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 989744128 bytes 2023-12-16T11:43:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 1058676736 bytes 2023-12-16T11:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 1073323008 bytes 2023-12-16T11:43:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 1095258112 bytes 2023-12-16T11:43:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 1102659584 bytes 2023-12-16T11:43:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 1142132736 bytes 2023-12-16T11:43:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1207358464 bytes 2023-12-16T11:43:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1227984896 bytes 2023-12-16T11:43:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1269955584 bytes 2023-12-16T11:43:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1323426816 bytes 2023-12-16T11:43:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1362330624 bytes 2023-12-16T11:43:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1447463936 bytes 2023-12-16T11:43:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1547275264 bytes 2023-12-16T11:43:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1576174592 bytes 2023-12-16T11:43:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1627454464 bytes 2023-12-16T11:43:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1736300544 bytes 2023-12-16T11:43:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1836317696 bytes 2023-12-16T11:43:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1896346624 bytes 2023-12-16T11:43:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1908206592 bytes 2023-12-16T11:43:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1985609728 bytes 2023-12-16T11:43:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 2002935808 bytes 2023-12-16T11:43:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 2058399744 bytes 2023-12-16T11:43:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 2060923904 bytes 2023-12-16T11:43:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 2100064256 bytes 2023-12-16T11:43:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 2172521472 bytes 2023-12-16T11:43:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 2210857984 bytes 2023-12-16T11:43:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 2261051392 bytes 2023-12-16T11:43:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 2271947776 bytes 2023-12-16T11:43:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 2309199872 bytes 2023-12-16T11:43:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 2398181376 bytes 2023-12-16T11:43:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 2424690688 bytes 2023-12-16T11:43:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 2436134912 bytes 2023-12-16T11:43:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 2468078592 bytes 2023-12-16T11:43:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 2559207424 bytes 2023-12-16T11:43:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2559834112 bytes 2023-12-16T11:43:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2598998016 bytes 2023-12-16T11:43:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2658495488 bytes 2023-12-16T11:43:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2661872640 bytes 2023-12-16T11:43:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2662516736 bytes 2023-12-16T11:43:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2683579392 bytes 2023-12-16T11:43:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2722782208 bytes 2023-12-16T11:43:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2776839168 bytes 2023-12-16T11:43:40.002 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2841806848 bytes 2023-12-16T11:43:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2865675264 bytes 2023-12-16T11:43:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2927357952 bytes 2023-12-16T11:43:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6800 IOs, 2929357824 bytes 2023-12-16T11:43:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6900 IOs, 2929995776 bytes 2023-12-16T11:43:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7000 IOs, 2995262464 bytes 2023-12-16T11:43:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7100 IOs, 3016503296 bytes 2023-12-16T11:43:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7200 IOs, 3066058752 bytes 2023-12-16T11:43:41.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7300 IOs, 3095747584 bytes 2023-12-16T11:43:41.523 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7400 IOs, 3119963136 bytes 2023-12-16T11:43:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7500 IOs, 3150246912 bytes 2023-12-16T11:43:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7600 IOs, 3224142848 bytes 2023-12-16T11:43:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7700 IOs, 3254569984 bytes 2023-12-16T11:43:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7800 IOs, 3260668928 bytes 2023-12-16T11:43:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:7900 IOs, 3281748992 bytes 2023-12-16T11:43:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:43:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:43:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:43:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:43:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-12-16T11:43:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:43:42.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_9 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_9 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-16T11:43:42.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:43:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=12c8225ba413 2023-12-16T11:43:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:43:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_9 present 12c8225ba413 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=12c8225ba413 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:43:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_9 2023-12-16T11:43:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:43:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:43:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_9 2023-12-16T11:43:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-16T11:43:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:43:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c8225ba413 2023-12-16T11:43:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c8225ba413 2023-12-16T11:43:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 12c8225ba413 -o 12c8225ba413 = 12c8225ba413 2023-12-16T11:43:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:43:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:43:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:43:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_9 2023-12-16T11:43:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-12-16T11:43:43.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:43:43.288+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa130800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-12-16T11:43:43.290 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:43:43.288+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa130800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-12-16T11:43:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:43:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_9 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_9 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-16T11:43:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:43:48.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:43:48.307+0000 7f2181893640 -1 rbd::mirror::ImageMap: 0x56442d1a8360 handle_peer_ack_remove: failed to remove global_image_id=54115782-4cc3-4ad5-803a-ca9c409d9c36 2023-12-16T11:43:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-16T11:43:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:43:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:43:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:43:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-12-16T11:43:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:43:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_9 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_9 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-12-16T11:43:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:43:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-16T11:43:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:43:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:43:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:43:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:43:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_9 2023-12-16T11:43:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:48.481 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-12-16T11:43:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-16T11:43:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:43:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:43:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_9 2023-12-16T11:43:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-16T11:43:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-16T11:43:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:43:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:43:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:43:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:43:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:43:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_9 2023-12-16T11:43:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-16T11:43:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-16T11:43:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:43:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:43:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:43:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:43:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:43:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_9 2023-12-16T11:43:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-16T11:43:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-16T11:43:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:43:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:43:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:43:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:43:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:43:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_9 2023-12-16T11:43:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:43:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:43:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:43:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-16T11:43:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-16T11:43:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:43:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:43:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:43:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:43:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:44:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_9 2023-12-16T11:44:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:44:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-16T11:44:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-16T11:44:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:44:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:44:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:44:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:44:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:44:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_9 2023-12-16T11:44:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_9 2023-12-16T11:44:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-16T11:44:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-16T11:44:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:44:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:44:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for i in `seq 1 10` 2023-12-16T11:44:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image=image_10 2023-12-16T11:44:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ create_image cluster2 mirror image_10 512M 2023-12-16T11:44:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local size=128 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -n 512M ']' 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ size=512M 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ shift 2023-12-16T11:44:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_10 2023-12-16T11:44:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ enable_mirror cluster2 mirror image_10 2023-12-16T11:44:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local mode=snapshot 2023-12-16T11:44:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-16T11:44:21.304 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring enabled 2023-12-16T11:44:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-16T11:44:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:rbd image 'image_10': 2023-12-16T11:44:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: size 512 MiB in 128 objects 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: order 22 (4 MiB objects) 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: snapshot_count: 1 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: id: 1a6338e6e3b6 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: block_name_prefix: rbd_data.1a6338e6e3b6 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: format: 2 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: features: layering 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: op_features: 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: flags: 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: create_timestamp: Sat Dec 16 11:44:20 2023 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: access_timestamp: Sat Dec 16 11:44:20 2023 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: modify_timestamp: Sat Dec 16 11:44:20 2023 2023-12-16T11:44:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring state: enabled 2023-12-16T11:44:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring mode: snapshot 2023-12-16T11:44:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring global id: 28305025-19e5-49aa-85d4-c9cd40b80b4c 2023-12-16T11:44:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout: mirroring primary: true 2023-12-16T11:44:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-16T11:44:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-12-16T11:44:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local duration=12 2023-12-16T11:44:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:44:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-16T11:44:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:100 IOs, 54103040 bytes 2023-12-16T11:44:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:200 IOs, 54752256 bytes 2023-12-16T11:44:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:300 IOs, 80838656 bytes 2023-12-16T11:44:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:400 IOs, 159680512 bytes 2023-12-16T11:44:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:500 IOs, 205936640 bytes 2023-12-16T11:44:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:600 IOs, 235277312 bytes 2023-12-16T11:44:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:700 IOs, 278020096 bytes 2023-12-16T11:44:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:800 IOs, 288824320 bytes 2023-12-16T11:44:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:900 IOs, 302971904 bytes 2023-12-16T11:44:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1000 IOs, 313872384 bytes 2023-12-16T11:44:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1100 IOs, 374129664 bytes 2023-12-16T11:44:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1200 IOs, 394853376 bytes 2023-12-16T11:44:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1300 IOs, 454128640 bytes 2023-12-16T11:44:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1400 IOs, 474692608 bytes 2023-12-16T11:44:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1500 IOs, 512605184 bytes 2023-12-16T11:44:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1600 IOs, 558751744 bytes 2023-12-16T11:44:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1700 IOs, 575633408 bytes 2023-12-16T11:44:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1800 IOs, 658345984 bytes 2023-12-16T11:44:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:1900 IOs, 717409280 bytes 2023-12-16T11:44:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2000 IOs, 728460288 bytes 2023-12-16T11:44:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2100 IOs, 729055232 bytes 2023-12-16T11:44:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2200 IOs, 765880320 bytes 2023-12-16T11:44:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2300 IOs, 796361728 bytes 2023-12-16T11:44:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2400 IOs, 832022528 bytes 2023-12-16T11:44:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2500 IOs, 896158720 bytes 2023-12-16T11:44:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2600 IOs, 949533696 bytes 2023-12-16T11:44:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2700 IOs, 965569536 bytes 2023-12-16T11:44:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2800 IOs, 997080064 bytes 2023-12-16T11:44:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:2900 IOs, 1025401856 bytes 2023-12-16T11:44:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3000 IOs, 1105767424 bytes 2023-12-16T11:44:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3100 IOs, 1130467328 bytes 2023-12-16T11:44:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3200 IOs, 1180545024 bytes 2023-12-16T11:44:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3300 IOs, 1201276928 bytes 2023-12-16T11:44:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3400 IOs, 1260633088 bytes 2023-12-16T11:44:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3500 IOs, 1330369536 bytes 2023-12-16T11:44:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3600 IOs, 1355552768 bytes 2023-12-16T11:44:27.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3700 IOs, 1380656128 bytes 2023-12-16T11:44:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3800 IOs, 1434125312 bytes 2023-12-16T11:44:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:3900 IOs, 1440396288 bytes 2023-12-16T11:44:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4000 IOs, 1463876608 bytes 2023-12-16T11:44:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4100 IOs, 1505603584 bytes 2023-12-16T11:44:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4200 IOs, 1528568832 bytes 2023-12-16T11:44:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4300 IOs, 1554473984 bytes 2023-12-16T11:44:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4400 IOs, 1582104576 bytes 2023-12-16T11:44:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4500 IOs, 1626174464 bytes 2023-12-16T11:44:28.866 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4600 IOs, 1642602496 bytes 2023-12-16T11:44:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4700 IOs, 1674382336 bytes 2023-12-16T11:44:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4800 IOs, 1694341120 bytes 2023-12-16T11:44:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:4900 IOs, 1747211264 bytes 2023-12-16T11:44:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5000 IOs, 1777391616 bytes 2023-12-16T11:44:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5100 IOs, 1806106624 bytes 2023-12-16T11:44:29.802 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9088258236677419) > weight (0.25) 2023-12-16T11:44:29.802 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-16T11:44:29.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-16T11:44:29.803 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-16T11:44:29.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:44:29.801+0000 7ff69ba32640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 117825) UID: 1000 2023-12-16T11:44:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5200 IOs, 1855716352 bytes 2023-12-16T11:44:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5300 IOs, 1881739264 bytes 2023-12-16T11:44:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5400 IOs, 1924896768 bytes 2023-12-16T11:44:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5500 IOs, 1925504000 bytes 2023-12-16T11:44:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5600 IOs, 1990456320 bytes 2023-12-16T11:44:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5700 IOs, 2013404160 bytes 2023-12-16T11:44:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5800 IOs, 2018646016 bytes 2023-12-16T11:44:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:5900 IOs, 2074289152 bytes 2023-12-16T11:44:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6000 IOs, 2139742208 bytes 2023-12-16T11:44:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6100 IOs, 2228142080 bytes 2023-12-16T11:44:31.458 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:44:31.455+0000 7f6f756e7640 -1 reset not still connected to 0x5577e9cc3ba0 2023-12-16T11:44:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6200 IOs, 2284261376 bytes 2023-12-16T11:44:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6300 IOs, 2326825984 bytes 2023-12-16T11:44:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6400 IOs, 2387426304 bytes 2023-12-16T11:44:32.151 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6500 IOs, 2408397824 bytes 2023-12-16T11:44:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6600 IOs, 2447026176 bytes 2023-12-16T11:44:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:6700 IOs, 2452194304 bytes 2023-12-16T11:44:32.535 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:44:32.533+0000 7f6f792c6640 -1 bdev(0x5577ac476e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-16T11:44:32.536 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6f792c6640 time 2023-12-16T11:44:32.534375+0000 2023-12-16T11:44:32.536 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5577a9349f01] 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:44:32.534+0000 7f6f792c6640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6f792c6640 time 2023-12-16T11:44:32.534375+0000 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5577a9349f01] 2023-12-16T11:44:32.537 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.538 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.538 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.538 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.538 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.538 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:*** Caught signal (Aborted) ** 2023-12-16T11:44:32.538 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: in thread 7f6f792c6640 thread_name:bstore_aio 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6f86254db0] 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f6f862a154c] 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: raise() 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: abort() 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5577a9349fc2] 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 7: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:2023-12-16T11:44:32.535+0000 7f6f792c6640 -1 *** Caught signal (Aborted) ** 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: in thread 7f6f792c6640 thread_name:bstore_aio 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.539 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6f86254db0] 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f6f862a154c] 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: raise() 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: abort() 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5577a9349fc2] 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 7: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-16T11:44:32.540 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.553 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: -30> 2023-12-16T11:44:32.533+0000 7f6f792c6640 -1 bdev(0x5577ac476e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-16T11:44:32.553 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: -14> 2023-12-16T11:44:32.534+0000 7f6f792c6640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6f792c6640 time 2023-12-16T11:44:32.534375+0000 2023-12-16T11:44:32.553 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5577a9349f01] 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 0> 2023-12-16T11:44:32.535+0000 7f6f792c6640 -1 *** Caught signal (Aborted) ** 2023-12-16T11:44:32.554 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: in thread 7f6f792c6640 thread_name:bstore_aio 2023-12-16T11:44:32.555 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.555 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.555 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6f86254db0] 2023-12-16T11:44:32.555 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f6f862a154c] 2023-12-16T11:44:32.555 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: raise() 2023-12-16T11:44:32.555 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: abort() 2023-12-16T11:44:32.555 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5577a9349fc2] 2023-12-16T11:44:32.556 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.556 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 7: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.556 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.556 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.556 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-16T11:44:32.556 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.564 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: -175> 2023-12-16T11:44:32.533+0000 7f6f792c6640 -1 bdev(0x5577ac476e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-16T11:44:32.564 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: -159> 2023-12-16T11:44:32.534+0000 7f6f792c6640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f6f792c6640 time 2023-12-16T11:44:32.534375+0000 2023-12-16T11:44:32.564 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-191-g6d933094/rpm/el9/BUILD/ceph-19.0.0-191-g6d933094/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-16T11:44:32.564 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.564 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x5577a9349f01] 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: -145> 2023-12-16T11:44:32.535+0000 7f6f792c6640 -1 *** Caught signal (Aborted) ** 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: in thread 7f6f792c6640 thread_name:bstore_aio 2023-12-16T11:44:32.565 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev) 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f6f86254db0] 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f6f862a154c] 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 3: raise() 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 4: abort() 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x5577a9349fc2] 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 6: (KernelDevice::_aio_thread()+0x10f6) [0x5577a9c9c6c6] 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 7: ceph-osd(+0xda5a71) [0x5577a9c7da71] 2023-12-16T11:44:32.566 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f6f8629f802] 2023-12-16T11:44:32.567 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f6f8623f450] 2023-12-16T11:44:32.567 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-16T11:44:32.567 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr: 2023-12-16T11:44:32.816 INFO:tasks.ceph.cluster2.osd.0.smithi160.stderr:daemon-helper: command failed with exit status 5 2023-12-16T11:44:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=124 2023-12-16T11:44:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:44:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 124 -eq 124 ']' 2023-12-16T11:44:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:44:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-12-16T11:44:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=present 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_10 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_10 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-16T11:44:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:44:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id=12c8e1fabcc 2023-12-16T11:44:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:44:33.403 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:44:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_10 present 12c8e1fabcc 2023-12-16T11:44:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:44:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=present 2023-12-16T11:44:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id=12c8e1fabcc 2023-12-16T11:44:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:44:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:44:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_10 2023-12-16T11:44:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:44:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:44:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_10 2023-12-16T11:44:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-16T11:44:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:44:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id=12c8e1fabcc 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 12c8e1fabcc 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -z 12c8e1fabcc -o 12c8e1fabcc = 12c8e1fabcc 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_state=present 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test present = present 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ disable_mirror cluster2 mirror image_10 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-12-16T11:44:34.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:44:34.793+0000 7f5cb833c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555f9ac21380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-16T11:44:34.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:44:34.793+0000 7f5cbeb49640 -1 rbd::mirror::ImageReplayer: 0x555f9a96bb80 [3/28305025-19e5-49aa-85d4-c9cd40b80b4c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-16T11:44:35.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:44:35.317+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa5efc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2023-12-16T11:44:35.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:44:35.317+0000 7f5eaf183640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55feaa5efc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=b3fc9f3f-6c2e-41a1-9b86-a18ef0448ca1, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2023-12-16T11:44:36.803 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-16T11:44:36.804 INFO:teuthology.orchestra.run:waiting for 600 2023-12-16T11:44:36.804 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-16T11:44:36.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-16T11:44:36.804 DEBUG:teuthology.orchestra.run.smithi160:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-16T11:44:36.807 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-16T11:44:36.808 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-12-16T11:44:36.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-16T11:44:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:Mirroring disabled 2023-12-16T11:44:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-16T11:44:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:44:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local state=deleted 2023-12-16T11:44:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:44:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local s 2023-12-16T11:44:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:44:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_10 2023-12-16T11:44:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:44:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:44:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_10 2023-12-16T11:44:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-16T11:44:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:44:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-16T11:44:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ image_id= 2023-12-16T11:44:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-12-16T11:44:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0.1 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local test_state=deleted 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image_id= 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_state=deleted 2023-12-16T11:44:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local current_image_id 2023-12-16T11:44:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ get_image_id cluster1 mirror image_10 2023-12-16T11:44:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:44:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:44:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local image=image_10 2023-12-16T11:44:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-12-16T11:44:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-12-16T11:44:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ current_image_id= 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n '' 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test deleted = deleted 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ purge_snapshots cluster2 mirror image_10 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2023-12-16T11:44:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-16T11:44:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:44:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:44:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_10 2023-12-16T11:44:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-16T11:44:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-16T11:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:44:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:44:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:44:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 1 2023-12-16T11:44:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_10 2023-12-16T11:44:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-16T11:44:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-16T11:44:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:44:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:44:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:44:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:44:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 2 2023-12-16T11:44:42.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-16T11:44:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_10 2023-12-16T11:44:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-16T11:44:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-16T11:44:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:44:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:44:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:44:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 4 2023-12-16T11:44:47.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-16T11:44:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_10 2023-12-16T11:44:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-16T11:44:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-16T11:44:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:44:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:44:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:44:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:44:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 8 2023-12-16T11:44:53.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-16T11:44:55.774 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5421105312470913) > weight (0.25) 2023-12-16T11:44:55.774 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.44892773667442) > weight (0.25) 2023-12-16T11:44:55.774 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9869954381818363) > weight (0.25) 2023-12-16T11:44:55.774 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8084581150332071) > weight (0.25) 2023-12-16T11:44:55.774 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-12-16T11:44:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_10 2023-12-16T11:44:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:44:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:44:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:44:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-16T11:44:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-16T11:44:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 0% complete...failed. 2023-12-16T11:44:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:rbd: error: image still has watchers 2023-12-16T11:44:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-16T11:44:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 4 8 16 32 2023-12-16T11:44:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 16 2023-12-16T11:44:58.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-16T11:45:03.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-16T11:45:09.335 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-16T11:45:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ remove_image cluster2 mirror image_10 2023-12-16T11:45:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:45:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local image=image_10 2023-12-16T11:45:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-16T11:45:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-16T11:45:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-16T11:45:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:45:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2023-12-16T11:45:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ tee -a /tmp/tmp.jvOAE3Umds/rbd-mirror.test.log 2023-12-16T11:45:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ date '+%F %T' 2023-12-16T11:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo 2023-12-16 11:45:12 TEST: check if removed 'images'\''' OMAP are removed 2023-12-16T11:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:2023-12-16 11:45:12 TEST: check if removed images' OMAP are removed 2023-12-16T11:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_in_omap cluster1 mirror 2023-12-16T11:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2023-12-16T11:45:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local obj_name=rbd_mirroring 2023-12-16T11:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local filter=status_global 2023-12-16T11:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-16T11:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:45:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local filter=status_global 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-16T11:45:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -c status_global 2023-12-16T11:45:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 = 0 2023-12-16T11:45:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:45:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:45:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 0 ']' 2023-12-16T11:45:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:45:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2023-12-16T11:45:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:45:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local obj_name=rbd_mirroring 2023-12-16T11:45:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local filter=image_ 2023-12-16T11:45:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-16T11:45:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:45:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:45:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2023-12-16T11:45:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:45:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local filter=image_ 2023-12-16T11:45:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-16T11:45:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:45:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-16T11:45:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -c image_ 2023-12-16T11:45:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 = 0 2023-12-16T11:45:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:45:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:45:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 0 ']' 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local obj_name=rbd_mirror_leader 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local filter=image_map 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:45:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:45:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-16T11:45:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:45:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirror_leader 2023-12-16T11:45:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local filter=image_map 2023-12-16T11:45:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-16T11:45:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster1 2023-12-16T11:45:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirror_leader 2023-12-16T11:45:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -c image_map 2023-12-16T11:45:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-12-16T11:45:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 = 0 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 0 ']' 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_image_in_omap cluster2 mirror 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local obj_name=rbd_mirroring 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local filter=status_global 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-16T11:45:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:45:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster2 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local filter=status_global 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster2 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-16T11:45:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -c status_global 2023-12-16T11:45:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 = 0 2023-12-16T11:45:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:45:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 0 ']' 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local obj_name=rbd_mirroring 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local filter=image_ 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:45:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:45:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2023-12-16T11:45:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster2 2023-12-16T11:45:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local filter=image_ 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster2 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -c image_ 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirroring 2023-12-16T11:45:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 = 0 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 0 ']' 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local pool=mirror 2023-12-16T11:45:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local obj_name=rbd_mirror_leader 2023-12-16T11:45:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local filter=image_map 2023-12-16T11:45:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-12-16T11:45:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ sleep 0 2023-12-16T11:45:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster2 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirror_leader 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local filter=image_map 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local cluster=cluster2 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ grep -c image_map 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local pool=mirror 2023-12-16T11:45:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ local obj_name=rbd_mirror_leader 2023-12-16T11:45:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2023-12-16T11:45:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test 0 = 0 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ error_code=0 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set -e 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 0 ']' 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 0 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ cleanup 0 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local error_code=0 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ set +e 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -ne 0 ']' 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -z '' ']' 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster instance 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:45:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-16T11:45:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror' does not exist 2023-12-16T11:45:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:45:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-16T11:45:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror' does not exist 2023-12-16T11:45:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:45:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-16T11:45:14.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-16T11:45:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror_parent' does not exist 2023-12-16T11:45:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ CEPH_ARGS= 2023-12-16T11:45:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-16T11:45:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.769+0000 7f2181893640 -1 rbd::mirror::LeaderWatcher: 0x56442cd3bb00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-12-16T11:45:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f217b086640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56442d3b8900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f217b086640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56442ceaeb40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f217b086640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x56442ceae000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f217a885640 -1 rbd::mirror::PoolWatcher: 0x56442bdbf860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f217a885640 -1 rbd::mirror::PoolWatcher: 0x56442bdbf0e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:15.773 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f217a885640 -1 rbd::mirror::PoolWatcher: 0x56442bdbe3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f2174879640 -1 rbd::mirror::PoolWatcher: 0x56442bdbfa40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f2174879640 -1 rbd::mirror::PoolWatcher: 0x56442bdbfc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f2174879640 -1 rbd::mirror::PoolWatcher: 0x56442d692000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f2181893640 -1 rbd::mirror::LeaderWatcher: 0x56442cd3bb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-16T11:45:15.774 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:15.770+0000 7f2181893640 -1 rbd::mirror::LeaderWatcher: 0x56442cd3bb00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-12-16T11:45:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:pool 'mirror_parent' does not exist 2023-12-16T11:45:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-16T11:45:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirrors cluster1 2023-12-16T11:45:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1 2023-12-16T11:45:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 0 3 2023-12-16T11:45:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster1:0 '' 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:0 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster1:1 '' 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:1 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster1:2 '' 2023-12-16T11:45:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:2 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster1:3 '' 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster1:3 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirrors cluster2 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:++ seq 0 3 2023-12-16T11:45:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster2:0 '' 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:0 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster2:1 '' 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:1 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster2:2 '' 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:2 2023-12-16T11:45:16.420 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ stop_mirror cluster2:3 '' 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local cluster=cluster2:3 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ local sig= 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test -n 1 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ return 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' -z 1 ']' 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ test '' = /tmp/tmp.jvOAE3Umds 2023-12-16T11:45:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ rm -Rf /tmp/tmp.jvOAE3Umds 2023-12-16T11:45:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ '[' 0 -eq 0 ']' 2023-12-16T11:45:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ echo OK 2023-12-16T11:45:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stdout:OK 2023-12-16T11:45:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+ exit 0 2023-12-16T11:45:16.423 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-16T11:45:16.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-12-16T11:45:16.491 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-12-16T11:45:16.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-12-16T11:45:16.804 DEBUG:teuthology.parallel:result is None 2023-12-16T11:45:16.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-16T11:45:16.831 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-16T11:45:16.832 DEBUG:teuthology.orchestra.run.smithi160:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-16T11:45:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b872d0640 -1 rbd::mirror::LeaderWatcher: 0x556997717b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-12-16T11:45:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b80ac3640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x556997db4fc0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b80ac3640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x556997bc0480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b802c2640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5569978c8900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b80ac3640 -1 rbd::mirror::PoolWatcher: 0x5569967a1680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:16.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b802c2640 -1 rbd::mirror::PoolWatcher: 0x5569967a14a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b802c2640 -1 rbd::mirror::PoolWatcher: 0x5569967a1860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b7a2b6640 -1 rbd::mirror::PoolWatcher: 0x5569967a1c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b7a2b6640 -1 rbd::mirror::PoolWatcher: 0x5569967a1a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.923+0000 7f1b7a2b6640 -1 rbd::mirror::PoolWatcher: 0x5569967a03c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.924+0000 7f1b872d0640 -1 rbd::mirror::LeaderWatcher: 0x556997717b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-16T11:45:16.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:16.924+0000 7f1b872d0640 -1 rbd::mirror::LeaderWatcher: 0x556997717b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-12-16T11:45:16.961 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-16T11:45:16.961 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-16T11:45:16.973 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-16T11:45:16.973 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 12 2023-12-16T11:45:16.973 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-16T11:45:16.973 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-16T11:45:17.002 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-16T11:45:17.002 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 15 2023-12-16T11:45:17.002 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-16T11:45:17.003 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.027 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-12-16T11:45:17.027 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.026+0000 7f6483607640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 125503) UID: 1000 2023-12-16T11:45:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.026+0000 7f6482e06640 -1 rbd::mirror::LeaderWatcher: 0x5648fa2c4300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.026+0000 7f6482605640 -1 rbd::mirror::InstanceWatcher: 0x5648f9ece000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6410 object: (2) No such file or directory 2023-12-16T11:45:17.028 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.026+0000 7f6482605640 -1 rbd::mirror::InstanceWatcher: 0x5648fa37f500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6410 object: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.026+0000 7f6482605640 -1 rbd::mirror::InstanceWatcher: 0x5648fa37e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6413 object: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f647c5f9640 -1 rbd::mirror::InstanceWatcher: 0x5648f9ece000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f647c5f9640 -1 rbd::mirror::InstanceWatcher: 0x5648fa37f500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f646e5dd640 -1 rbd::mirror::InstanceWatcher: 0x5648fa37e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6475dec640 -1 rbd::mirror::MirrorStatusUpdater 0x5648f9f225a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6475dec640 -1 rbd::mirror::MirrorStatusUpdater 0x5648f9f22360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6482e06640 -1 rbd::mirror::NamespaceReplayer: 0x5648f9ec6820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6482e06640 -1 rbd::mirror::NamespaceReplayer: 0x5648f9ec69c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f64685d1640 -1 rbd::mirror::MirrorStatusUpdater 0x5648f9f226c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6482605640 -1 rbd::mirror::NamespaceReplayer: 0x5648fa2ca4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f647bdf8640 -1 rbd::mirror::MirrorStatusUpdater 0x5648f9f22000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f647bdf8640 -1 rbd::mirror::MirrorStatusUpdater 0x5648f9f22120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6482e06640 -1 rbd::mirror::NamespaceReplayer: 0x5648f9ec6820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6482e06640 -1 rbd::mirror::NamespaceReplayer: 0x5648f9ec69c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f646edde640 -1 rbd::mirror::MirrorStatusUpdater 0x5648fa14e6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.027+0000 7f6482605640 -1 rbd::mirror::NamespaceReplayer: 0x5648fa2ca4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.029+0000 7f6482e06640 -1 rbd::mirror::LeaderWatcher: 0x5648f9ca3b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.031 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.029+0000 7f6482e06640 -1 rbd::mirror::InstanceWatcher: 0x5648f9bdb880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6410 object: (2) No such file or directory 2023-12-16T11:45:17.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.030+0000 7f647c5f9640 -1 rbd::mirror::InstanceWatcher: 0x5648f9bdb880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.030+0000 7f64755eb640 -1 rbd::mirror::MirrorStatusUpdater 0x5648f9caa5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.030+0000 7f6482e06640 -1 rbd::mirror::NamespaceReplayer: 0x5648f9ec6000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.030+0000 7f647bdf8640 -1 rbd::mirror::MirrorStatusUpdater 0x5648f9c50b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.032 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.030+0000 7f6482605640 -1 rbd::mirror::NamespaceReplayer: 0x5648f9ec6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.043 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2023-12-16T11:45:17.044 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.053 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-12-16T11:45:17.053 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.083+0000 7f5cbf34a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 105236) UID: 1000 2023-12-16T11:45:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.083+0000 7f5caa320640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x555f9a5fa900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbe348640 -1 rbd::mirror::InstanceWatcher: 0x555f9a2fa380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5281 object: (2) No such file or directory 2023-12-16T11:45:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbe348640 -1 rbd::mirror::InstanceWatcher: 0x555f9a2faa80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5281 object: (2) No such file or directory 2023-12-16T11:45:17.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5caab21640 -1 rbd::mirror::PoolWatcher: 0x555f9900e3c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cb7b3b640 -1 rbd::mirror::InstanceWatcher: 0x555f9a2fa380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cb7b3b640 -1 rbd::mirror::InstanceWatcher: 0x555f9a2faa80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5ca3b13640 -1 rbd::mirror::PoolWatcher: 0x555f9900eb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cb1b2f640 -1 rbd::mirror::MirrorStatusUpdater 0x555f9a204a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cb1b2f640 -1 rbd::mirror::MirrorStatusUpdater 0x555f9a2045a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbeb49640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a1b4820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbeb49640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a1b49c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cb7b3b640 -1 rbd::mirror::MirrorStatusUpdater 0x555f9a204240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cb7b3b640 -1 rbd::mirror::MirrorStatusUpdater 0x555f9a204360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbe348640 -1 rbd::mirror::LeaderWatcher: 0x555f9a5f4300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbe348640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a1b4820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbe348640 -1 rbd::mirror::LeaderWatcher: 0x555f9a5f4300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.084+0000 7f5cbe348640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a1b49c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.085+0000 7f5cbeb49640 -1 rbd::mirror::LeaderWatcher: 0x555f9a5f4300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.085+0000 7f5cbeb49640 -1 rbd::mirror::InstanceWatcher: 0x555f9a1c5c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5283 object: (2) No such file or directory 2023-12-16T11:45:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.085+0000 7f5caa320640 -1 rbd::mirror::InstanceWatcher: 0x555f9a1c5c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.085+0000 7f5ca4314640 -1 rbd::mirror::MirrorStatusUpdater 0x555f9a41a7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.085+0000 7f5cbeb49640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a5f84e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.086+0000 7f5caab21640 -1 rbd::mirror::MirrorStatusUpdater 0x555f9a3d67e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.086+0000 7f5cbe348640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a5f84e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.088+0000 7f5cbeb49640 -1 rbd::mirror::LeaderWatcher: 0x555f99f87b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.088+0000 7f5cbeb49640 -1 rbd::mirror::InstanceWatcher: 0x555f99e7b880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5281 object: (2) No such file or directory 2023-12-16T11:45:17.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.088+0000 7f5cb7b3b640 -1 rbd::mirror::InstanceWatcher: 0x555f99e7b880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.088+0000 7f5cb132e640 -1 rbd::mirror::MirrorStatusUpdater 0x555f99f347e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.088+0000 7f5cbeb49640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a1b4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.089+0000 7f5cb833c640 -1 rbd::mirror::MirrorStatusUpdater 0x555f99f00b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.089+0000 7f5cbe348640 -1 rbd::mirror::NamespaceReplayer: 0x555f9a1b4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2023-12-16T11:45:17.103 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.113 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-12-16T11:45:17.113 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.142+0000 7f2182094640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 123374) UID: 1000 2023-12-16T11:45:17.145 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.142+0000 7f2181893640 -1 rbd::mirror::LeaderWatcher: 0x56442d37a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.143+0000 7f2181893640 -1 rbd::mirror::InstanceWatcher: 0x56442d062000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6278 object: (2) No such file or directory 2023-12-16T11:45:17.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.143+0000 7f2181893640 -1 rbd::mirror::InstanceWatcher: 0x56442d062700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6278 object: (2) No such file or directory 2023-12-16T11:45:17.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.143+0000 7f2181092640 -1 rbd::mirror::InstanceWatcher: 0x56442d2c1180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6281 object: (2) No such file or directory 2023-12-16T11:45:17.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.143+0000 7f217b086640 -1 rbd::mirror::InstanceWatcher: 0x56442d062000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.143+0000 7f217b086640 -1 rbd::mirror::InstanceWatcher: 0x56442d062700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.146 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.143+0000 7f216d06a640 -1 rbd::mirror::InstanceWatcher: 0x56442d2c1180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2174078640 -1 rbd::mirror::MirrorStatusUpdater 0x56442cf86360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2174078640 -1 rbd::mirror::MirrorStatusUpdater 0x56442cf86480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f216705e640 -1 rbd::mirror::MirrorStatusUpdater 0x56442d1a87e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2181893640 -1 rbd::mirror::NamespaceReplayer: 0x56442cf329c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2181893640 -1 rbd::mirror::NamespaceReplayer: 0x56442cf32b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2181893640 -1 rbd::mirror::NamespaceReplayer: 0x56442d3a2340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f217a885640 -1 rbd::mirror::MirrorStatusUpdater 0x56442cf86000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f217a885640 -1 rbd::mirror::MirrorStatusUpdater 0x56442cf86120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2181092640 -1 rbd::mirror::NamespaceReplayer: 0x56442cf329c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f216d06a640 -1 rbd::mirror::MirrorStatusUpdater 0x56442d1446c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2181092640 -1 rbd::mirror::NamespaceReplayer: 0x56442cf32b60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.144+0000 7f2181092640 -1 rbd::mirror::NamespaceReplayer: 0x56442d3a2340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.146+0000 7f2181893640 -1 rbd::mirror::LeaderWatcher: 0x56442cd3bb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.146+0000 7f2181893640 -1 rbd::mirror::InstanceWatcher: 0x56442cc37880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6278 object: (2) No such file or directory 2023-12-16T11:45:17.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.146+0000 7f217b086640 -1 rbd::mirror::InstanceWatcher: 0x56442cc37880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.147+0000 7f2174879640 -1 rbd::mirror::MirrorStatusUpdater 0x56442cd226c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.147+0000 7f2181893640 -1 rbd::mirror::NamespaceReplayer: 0x56442cf32000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.147+0000 7f217b086640 -1 rbd::mirror::MirrorStatusUpdater 0x56442cc88a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.149 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.147+0000 7f2181092640 -1 rbd::mirror::NamespaceReplayer: 0x56442cf32000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.160 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-12-16T11:45:17.160 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.171 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-12-16T11:45:17.171 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.201 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.199+0000 7f5cdcb95640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 128497) UID: 1000 2023-12-16T11:45:17.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cdbb93640 -1 rbd::mirror::InstanceWatcher: 0x557c1c64e700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6722 object: (2) No such file or directory 2023-12-16T11:45:17.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cdbb93640 -1 rbd::mirror::InstanceWatcher: 0x557c1c5d4e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6720 object: (2) No such file or directory 2023-12-16T11:45:17.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cdbb93640 -1 rbd::mirror::InstanceWatcher: 0x557c1c64e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6720 object: (2) No such file or directory 2023-12-16T11:45:17.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cc7b6b640 -1 rbd::mirror::InstanceWatcher: 0x557c1c64e700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.202 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cd5386640 -1 rbd::mirror::InstanceWatcher: 0x557c1c5d4e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cd5386640 -1 rbd::mirror::InstanceWatcher: 0x557c1c64e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cceb79640 -1 rbd::mirror::MirrorStatusUpdater 0x557c1c622360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cceb79640 -1 rbd::mirror::MirrorStatusUpdater 0x557c1c6225a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cdc394640 -1 rbd::mirror::NamespaceReplayer: 0x557c1c5d2820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.200+0000 7f5cdc394640 -1 rbd::mirror::NamespaceReplayer: 0x557c1c5d29c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.201+0000 7f5cd5b87640 -1 rbd::mirror::MirrorStatusUpdater 0x557c1c622000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.201+0000 7f5cd5b87640 -1 rbd::mirror::MirrorStatusUpdater 0x557c1c622120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.201+0000 7f5cdbb93640 -1 rbd::mirror::NamespaceReplayer: 0x557c1c5d2820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.201+0000 7f5cdbb93640 -1 rbd::mirror::NamespaceReplayer: 0x557c1c5d29c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.202+0000 7f5cdbb93640 -1 rbd::mirror::LeaderWatcher: 0x557c1c3a7b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.202+0000 7f5cdbb93640 -1 rbd::mirror::InstanceWatcher: 0x557c1c1d9880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6720 object: (2) No such file or directory 2023-12-16T11:45:17.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.202+0000 7f5cd5b87640 -1 rbd::mirror::InstanceWatcher: 0x557c1c1d9880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.203+0000 7f5cceb79640 -1 rbd::mirror::MirrorStatusUpdater 0x557c1c38c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.203+0000 7f5cdbb93640 -1 rbd::mirror::NamespaceReplayer: 0x557c1c5d2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.203+0000 7f5cd5386640 -1 rbd::mirror::MirrorStatusUpdater 0x557c1c336b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.203+0000 7f5cdc394640 -1 rbd::mirror::NamespaceReplayer: 0x557c1c5d2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.216 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2023-12-16T11:45:17.216 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.226 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-12-16T11:45:17.226 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.257 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.255+0000 7fed77afc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 128391) UID: 1000 2023-12-16T11:45:17.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.256+0000 7fed76afa640 -1 rbd::mirror::LeaderWatcher: 0x55d95a888300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.256+0000 7fed772fb640 -1 rbd::mirror::InstanceWatcher: 0x55d95a482e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6081 object: (2) No such file or directory 2023-12-16T11:45:17.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.256+0000 7fed772fb640 -1 rbd::mirror::InstanceWatcher: 0x55d95a483880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6081 object: (2) No such file or directory 2023-12-16T11:45:17.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.256+0000 7fed772fb640 -1 rbd::mirror::InstanceWatcher: 0x55d95a649500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6084 object: (2) No such file or directory 2023-12-16T11:45:17.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.256+0000 7fed70aee640 -1 rbd::mirror::InstanceWatcher: 0x55d95a482e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.256+0000 7fed702ed640 -1 rbd::mirror::InstanceWatcher: 0x55d95a483880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed6a2e1640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a4bc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed62ad2640 -1 rbd::mirror::InstanceWatcher: 0x55d95a649500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed6a2e1640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a4bc360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed772fb640 -1 rbd::mirror::NamespaceReplayer: 0x55d95a4ba000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed772fb640 -1 rbd::mirror::NamespaceReplayer: 0x55d95a4ba1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed702ed640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a4bc000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed702ed640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a4bc120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed76afa640 -1 rbd::mirror::NamespaceReplayer: 0x55d95a4ba000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed76afa640 -1 rbd::mirror::NamespaceReplayer: 0x55d95a4ba1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed5c2c5640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a7007e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed772fb640 -1 rbd::mirror::NamespaceReplayer: 0x55d95a93e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed632d3640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a6985a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.257+0000 7fed76afa640 -1 rbd::mirror::NamespaceReplayer: 0x55d95a93e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.259+0000 7fed772fb640 -1 rbd::mirror::LeaderWatcher: 0x55d95a25bb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.259+0000 7fed772fb640 -1 rbd::mirror::InstanceWatcher: 0x55d95a043880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6081 object: (2) No such file or directory 2023-12-16T11:45:17.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.260+0000 7fed70aee640 -1 rbd::mirror::InstanceWatcher: 0x55d95a043880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.260+0000 7fed6a2e1640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a1ee7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.260+0000 7fed772fb640 -1 rbd::mirror::NamespaceReplayer: 0x55d95acb01a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.260+0000 7fed702ed640 -1 rbd::mirror::MirrorStatusUpdater 0x55d95a1a8b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi160.stderr:2023-12-16T11:45:17.260+0000 7fed76afa640 -1 rbd::mirror::NamespaceReplayer: 0x55d95acb01a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2023-12-16T11:45:17.273 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.285 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-12-16T11:45:17.286 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.312+0000 7f1b87ad1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 124035) UID: 1000 2023-12-16T11:45:17.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b732a8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5569983a7680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b872d0640 -1 rbd::mirror::InstanceWatcher: 0x556997973880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5939 object: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b872d0640 -1 rbd::mirror::InstanceWatcher: 0x5569979db500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5939 object: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b72aa7640 -1 rbd::mirror::PoolWatcher: 0x556997f125a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b6c29a640 -1 rbd::mirror::PoolWatcher: 0x556997f121e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b80ac3640 -1 rbd::mirror::InstanceWatcher: 0x556997973880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b80ac3640 -1 rbd::mirror::InstanceWatcher: 0x5569979db500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b872d0640 -1 rbd::mirror::LeaderWatcher: 0x556997cf0300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b872d0640 -1 rbd::mirror::LeaderWatcher: 0x556997cf0300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b79ab5640 -1 rbd::mirror::MirrorStatusUpdater 0x5569979a6360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b79ab5640 -1 rbd::mirror::MirrorStatusUpdater 0x5569979a65a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b86acf640 -1 rbd::mirror::NamespaceReplayer: 0x556997958820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b86acf640 -1 rbd::mirror::NamespaceReplayer: 0x5569979589c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.317 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b872d0640 -1 rbd::mirror::LeaderWatcher: 0x556997cf0300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b802c2640 -1 rbd::mirror::MirrorStatusUpdater 0x5569979a6000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b872d0640 -1 rbd::mirror::NamespaceReplayer: 0x556997958820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b80ac3640 -1 rbd::mirror::MirrorStatusUpdater 0x5569979a6120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.313+0000 7f1b86acf640 -1 rbd::mirror::NamespaceReplayer: 0x5569979589c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.314+0000 7f1b86acf640 -1 rbd::mirror::InstanceWatcher: 0x55699795c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5941 object: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.314+0000 7f1b72aa7640 -1 rbd::mirror::InstanceWatcher: 0x55699795c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.314+0000 7f1b6ca9b640 -1 rbd::mirror::MirrorStatusUpdater 0x556997bd47e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.314+0000 7f1b86acf640 -1 rbd::mirror::NamespaceReplayer: 0x556997cf6680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.314+0000 7f1b732a8640 -1 rbd::mirror::MirrorStatusUpdater 0x556997b907e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.314+0000 7f1b872d0640 -1 rbd::mirror::NamespaceReplayer: 0x556997cf6680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.317+0000 7f1b86acf640 -1 rbd::mirror::LeaderWatcher: 0x556997717b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.319 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.317+0000 7f1b86acf640 -1 rbd::mirror::InstanceWatcher: 0x556997537880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5939 object: (2) No such file or directory 2023-12-16T11:45:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.317+0000 7f1b80ac3640 -1 rbd::mirror::InstanceWatcher: 0x556997537880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.317+0000 7f1b79ab5640 -1 rbd::mirror::MirrorStatusUpdater 0x5569976fe6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.317+0000 7f1b86acf640 -1 rbd::mirror::NamespaceReplayer: 0x556997958000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.318+0000 7f1b802c2640 -1 rbd::mirror::MirrorStatusUpdater 0x556997696b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi160.stderr:2023-12-16T11:45:17.318+0000 7f1b872d0640 -1 rbd::mirror::NamespaceReplayer: 0x556997958000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2023-12-16T11:45:17.331 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.342 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-12-16T11:45:17.342 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.370+0000 7f0bc491f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 129193) UID: 1000 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bc391d640 -1 rbd::mirror::InstanceWatcher: 0x55f5c5f1a700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6114 object: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bc391d640 -1 rbd::mirror::InstanceWatcher: 0x55f5c63ca700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6114 object: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bb00f6640 -1 rbd::mirror::InstanceWatcher: 0x55f5c63a6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bbd911640 -1 rbd::mirror::InstanceWatcher: 0x55f5c5f1a700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bbd911640 -1 rbd::mirror::InstanceWatcher: 0x55f5c63ca700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bb6903640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5c5f6e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bb6903640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5c5f6e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bc391d640 -1 rbd::mirror::NamespaceReplayer: 0x55f5c5f18820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bc391d640 -1 rbd::mirror::NamespaceReplayer: 0x55f5c5f189c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bbd110640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5c5f6e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bbd110640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5c5f6e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.374 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bc411e640 -1 rbd::mirror::NamespaceReplayer: 0x55f5c5f18820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.371+0000 7f0bc411e640 -1 rbd::mirror::NamespaceReplayer: 0x55f5c5f189c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.373+0000 7f0bc411e640 -1 rbd::mirror::LeaderWatcher: 0x55f5c5cebb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.373+0000 7f0bc411e640 -1 rbd::mirror::InstanceWatcher: 0x55f5c5be7880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6114 object: (2) No such file or directory 2023-12-16T11:45:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.374+0000 7f0bbd110640 -1 rbd::mirror::InstanceWatcher: 0x55f5c5be7880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.374+0000 7f0bb6903640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5c5cd46c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.374+0000 7f0bc411e640 -1 rbd::mirror::NamespaceReplayer: 0x55f5c5f18000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.376 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.374+0000 7f0bbd110640 -1 rbd::mirror::MirrorStatusUpdater 0x55f5c686cb40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi160.stderr:2023-12-16T11:45:17.374+0000 7f0bc391d640 -1 rbd::mirror::NamespaceReplayer: 0x55f5c5f18000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2023-12-16T11:45:17.387 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-16T11:45:17.402 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-12-16T11:45:17.402 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:45:17.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.426+0000 7f5ebc99e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 104018) UID: 1000 2023-12-16T11:45:17.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.426+0000 7f5ebc19d640 -1 rbd::mirror::LeaderWatcher: 0x55fea995c300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5ebc19d640 -1 rbd::mirror::InstanceWatcher: 0x55fea95f2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4808 object: (2) No such file or directory 2023-12-16T11:45:17.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5ebc19d640 -1 rbd::mirror::InstanceWatcher: 0x55fea9571880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4808 object: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5ebb99c640 -1 rbd::mirror::InstanceWatcher: 0x55fea9571180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4812 object: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5eb5990640 -1 rbd::mirror::InstanceWatcher: 0x55fea95f2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5eb5990640 -1 rbd::mirror::InstanceWatcher: 0x55fea9571880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5ea7974640 -1 rbd::mirror::InstanceWatcher: 0x55fea9571180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5eaf183640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea95aa5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5ebb99c640 -1 rbd::mirror::NamespaceReplayer: 0x55fea954e820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5eae982640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea95aa7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.427+0000 7f5ebc19d640 -1 rbd::mirror::NamespaceReplayer: 0x55fea954e9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5ea1968640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea97fe5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5ebb99c640 -1 rbd::mirror::NamespaceReplayer: 0x55fea99624e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5eb518f640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea95aa240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5ebb99c640 -1 rbd::mirror::NamespaceReplayer: 0x55fea954e820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5eb5990640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea95aa360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5ebc19d640 -1 rbd::mirror::NamespaceReplayer: 0x55fea954e9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5ea8175640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea97d8900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.428+0000 7f5ebb99c640 -1 rbd::mirror::NamespaceReplayer: 0x55fea99624e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.435 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.430+0000 7f5ebc19d640 -1 rbd::mirror::LeaderWatcher: 0x55fea9307b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-16T11:45:17.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.430+0000 7f5ebc19d640 -1 rbd::mirror::InstanceWatcher: 0x55fea9223880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4808 object: (2) No such file or directory 2023-12-16T11:45:17.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.431+0000 7f5eb5990640 -1 rbd::mirror::InstanceWatcher: 0x55fea9223880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-16T11:45:17.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.431+0000 7f5eae982640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea93125a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.431+0000 7f5ebc19d640 -1 rbd::mirror::NamespaceReplayer: 0x55fea954e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.431+0000 7f5eb518f640 -1 rbd::mirror::MirrorStatusUpdater 0x55fea92c6b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-16T11:45:17.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi160.stderr:2023-12-16T11:45:17.431+0000 7f5ebb99c640 -1 rbd::mirror::NamespaceReplayer: 0x55fea954e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-16T11:45:17.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2023-12-16T11:45:17.447 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-16T11:45:17.460 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-16T11:45:17.460 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:17.762 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:17.762 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:17.774 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1450,"stamp":"2023-12-16T11:45:16.292416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4271300,"kb_used_data":1044816,"kb_used_omap":293,"kb_used_meta":3226074,"kb_avail":278844220,"statfs":{"total":289910292480,"available":285536481280,"internally_reserved":0,"allocated":1069891584,"data_stored":1065310164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3303500218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536846390,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-110157,"num_read_kb":-39659723,"num_write":-420816,"num_write_kb":-154900233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.059478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":395,"reported_epoch":377,"state":"active+undersized","last_fresh":"2023-12-16T11:44:34.333462+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:44:34.333462+0000","last_peered":"2023-12-16T11:44:34.333462+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:44:34.333462+0000","last_undegraded":"2023-12-16T11:44:34.333462+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":395,"reported_epoch":377,"state":"active+undersized","last_fresh":"2023-12-16T11:44:34.322075+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:44:34.322075+0000","last_peered":"2023-12-16T11:44:34.322075+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:44:34.322075+0000","last_undegraded":"2023-12-16T11:44:34.322075+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":396,"reported_epoch":378,"state":"active+undersized","last_fresh":"2023-12-16T11:44:35.304631+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:44:35.304631+0000","last_peered":"2023-12-16T11:44:35.304631+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:44:35.304631+0000","last_undegraded":"2023-12-16T11:44:35.304631+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":396,"reported_epoch":378,"state":"active+undersized","last_fresh":"2023-12-16T11:44:35.302890+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:44:35.302890+0000","last_peered":"2023-12-16T11:44:35.302890+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:44:35.302890+0000","last_undegraded":"2023-12-16T11:44:35.302890+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":378,"state":"active+undersized","last_fresh":"2023-12-16T11:44:35.302486+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:44:35.302486+0000","last_peered":"2023-12-16T11:44:35.302486+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:44:35.302486+0000","last_undegraded":"2023-12-16T11:44:35.302486+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":392,"reported_epoch":378,"state":"active+clean","last_fresh":"2023-12-16T11:44:35.316826+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:44:35.316826+0000","last_peered":"2023-12-16T11:44:35.316826+0000","last_clean":"2023-12-16T11:44:35.316826+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:44:35.316826+0000","last_undegraded":"2023-12-16T11:44:35.316826+0000","last_fullsized":"2023-12-16T11:44:35.316826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":392,"reported_epoch":377,"state":"active+clean","last_fresh":"2023-12-16T11:44:34.299533+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:44:34.299533+0000","last_peered":"2023-12-16T11:44:34.299533+0000","last_clean":"2023-12-16T11:44:34.299533+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:44:34.299533+0000","last_undegraded":"2023-12-16T11:44:34.299533+0000","last_fullsized":"2023-12-16T11:44:34.299533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":392,"reported_epoch":378,"state":"active+clean","last_fresh":"2023-12-16T11:44:35.302099+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:44:35.302099+0000","last_peered":"2023-12-16T11:44:35.302099+0000","last_clean":"2023-12-16T11:44:35.302099+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:44:35.302099+0000","last_undegraded":"2023-12-16T11:44:35.302099+0000","last_fullsized":"2023-12-16T11:44:35.302099+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":526,"reported_epoch":377,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:44:34.308604+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:44:34.308604+0000","last_peered":"2023-12-16T11:44:34.308604+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:44:34.308604+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1117272,"kb_used_data":293648,"kb_used_omap":90,"kb_used_meta":823525,"kb_avail":93254568,"statfs":{"total":96636764160,"available":95492677632,"internally_reserved":0,"allocated":300695552,"data_stored":299235031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":843290204},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.27000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738802,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1352720,"kb_used_data":338568,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93019120,"statfs":{"total":96636764160,"available":95251578880,"internally_reserved":0,"allocated":346693632,"data_stored":345236692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.158},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.218}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:17.775 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:18.049 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:18.050 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:18.062 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1450,"stamp":"2023-12-16T11:45:16.292416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4271300,"kb_used_data":1044816,"kb_used_omap":293,"kb_used_meta":3226074,"kb_avail":278844220,"statfs":{"total":289910292480,"available":285536481280,"internally_reserved":0,"allocated":1069891584,"data_stored":1065310164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3303500218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536846390,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-110157,"num_read_kb":-39659723,"num_write":-420816,"num_write_kb":-154900233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.059478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":395,"reported_epoch":377,"state":"active+undersized","last_fresh":"2023-12-16T11:44:34.333462+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:44:34.333462+0000","last_peered":"2023-12-16T11:44:34.333462+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:44:34.333462+0000","last_undegraded":"2023-12-16T11:44:34.333462+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":395,"reported_epoch":377,"state":"active+undersized","last_fresh":"2023-12-16T11:44:34.322075+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:44:34.322075+0000","last_peered":"2023-12-16T11:44:34.322075+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:44:34.322075+0000","last_undegraded":"2023-12-16T11:44:34.322075+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":396,"reported_epoch":378,"state":"active+undersized","last_fresh":"2023-12-16T11:44:35.304631+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:44:35.304631+0000","last_peered":"2023-12-16T11:44:35.304631+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:44:35.304631+0000","last_undegraded":"2023-12-16T11:44:35.304631+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":396,"reported_epoch":378,"state":"active+undersized","last_fresh":"2023-12-16T11:44:35.302890+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:44:35.302890+0000","last_peered":"2023-12-16T11:44:35.302890+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:44:35.302890+0000","last_undegraded":"2023-12-16T11:44:35.302890+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":378,"state":"active+undersized","last_fresh":"2023-12-16T11:44:35.302486+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:44:35.302486+0000","last_peered":"2023-12-16T11:44:35.302486+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:44:35.302486+0000","last_undegraded":"2023-12-16T11:44:35.302486+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":392,"reported_epoch":378,"state":"active+clean","last_fresh":"2023-12-16T11:44:35.316826+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:44:35.316826+0000","last_peered":"2023-12-16T11:44:35.316826+0000","last_clean":"2023-12-16T11:44:35.316826+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:44:35.316826+0000","last_undegraded":"2023-12-16T11:44:35.316826+0000","last_fullsized":"2023-12-16T11:44:35.316826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":392,"reported_epoch":377,"state":"active+clean","last_fresh":"2023-12-16T11:44:34.299533+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:44:34.299533+0000","last_peered":"2023-12-16T11:44:34.299533+0000","last_clean":"2023-12-16T11:44:34.299533+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:44:34.299533+0000","last_undegraded":"2023-12-16T11:44:34.299533+0000","last_fullsized":"2023-12-16T11:44:34.299533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":392,"reported_epoch":378,"state":"active+clean","last_fresh":"2023-12-16T11:44:35.302099+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:44:35.302099+0000","last_peered":"2023-12-16T11:44:35.302099+0000","last_clean":"2023-12-16T11:44:35.302099+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:44:35.302099+0000","last_undegraded":"2023-12-16T11:44:35.302099+0000","last_fullsized":"2023-12-16T11:44:35.302099+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":526,"reported_epoch":377,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:44:34.308604+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:44:34.308604+0000","last_peered":"2023-12-16T11:44:34.308604+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:44:34.308604+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1117272,"kb_used_data":293648,"kb_used_omap":90,"kb_used_meta":823525,"kb_avail":93254568,"statfs":{"total":96636764160,"available":95492677632,"internally_reserved":0,"allocated":300695552,"data_stored":299235031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":843290204},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.27000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738802,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1352720,"kb_used_data":338568,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93019120,"statfs":{"total":96636764160,"available":95251578880,"internally_reserved":0,"allocated":346693632,"data_stored":345236692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.158},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.218}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:18.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:18.338 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:18.339 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:18.351 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1451,"stamp":"2023-12-16T11:45:18.236125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536846390,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-110157,"num_read_kb":-39659723,"num_write":-420816,"num_write_kb":-154900233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002407"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.53100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:18.352 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:18.353 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:18.353 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:18.353 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:18.353 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 528, 'reported_epoch': 379, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:14.291942+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:14.291942+0000', 'last_peered': '2023-12-16T11:45:14.291942+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:14.291942+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:18.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:18.694 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:18.694 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4/20 objects degraded (20.000%), 3 pgs degraded","count":3},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2156,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":44},{"state_name":"active+clean","count":26},{"state_name":"active+undersized+degraded","count":3}],"num_pgs":73,"num_pools":3,"num_objects":10,"data_bytes":590387,"bytes_used":4373811200,"bytes_avail":285536481280,"bytes_total":289910292480,"degraded_objects":4,"degraded_total":20,"degraded_ratio":0.20000000000000001},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":37,"modified":"2023-12-16T11:43:32.205222+0000","services":{"rbd-mirror":{"daemons":{"summary":"","5279":{"start_epoch":24,"start_stamp":"2023-12-16T11:29:04.657295+0000","gid":5279,"addr":"172.21.15.160:0/1527683113","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)","ceph_version_short":"19.0.0-191-g6d933094","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi160","id":"mirror.2","instance_id":"5279","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 28 20:35:49 UTC 2023","kernel_version":"5.14.0-391.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488632","os":"Linux"},"task_status":{}},"6275":{"start_epoch":30,"start_stamp":"2023-12-16T11:37:02.773887+0000","gid":6275,"addr":"172.21.15.160:0/1336945750","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)","ceph_version_short":"19.0.0-191-g6d933094","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi160","id":"mirror.1","instance_id":"6275","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 28 20:35:49 UTC 2023","kernel_version":"5.14.0-391.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488632","os":"Linux"},"task_status":{}},"6407":{"start_epoch":32,"start_stamp":"2023-12-16T11:39:09.743727+0000","gid":6407,"addr":"172.21.15.160:0/2646921380","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)","ceph_version_short":"19.0.0-191-g6d933094","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi160","id":"mirror.3","instance_id":"6407","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 28 20:35:49 UTC 2023","kernel_version":"5.14.0-391.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488632","os":"Linux"},"task_status":{}},"6717":{"start_epoch":37,"start_stamp":"2023-12-16T11:43:31.899079+0000","gid":6717,"addr":"172.21.15.160:0/3019548472","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)","ceph_version_short":"19.0.0-191-g6d933094","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","hostname":"smithi160","id":"mirror.0","instance_id":"6717","kernel_description":"#1 SMP PREEMPT_DYNAMIC Tue Nov 28 20:35:49 UTC 2023","kernel_version":"5.14.0-391.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488632","os":"Linux"},"task_status":{}}}}}},"progress_events":{}} 2023-12-16T11:45:18.706 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:18.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:18.982 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:18.982 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:18.994 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1451,"stamp":"2023-12-16T11:45:18.236125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536846390,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-110157,"num_read_kb":-39659723,"num_write":-420816,"num_write_kb":-154900233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002407"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.53100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:20.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-16T11:45:21.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:22.283 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:22.284 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:22.296 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1453,"stamp":"2023-12-16T11:45:22.236906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-251658294,"num_objects":-95,"num_object_clones":0,"num_object_copies":-190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-95,"num_whiteouts":0,"num_read":-110288,"num_read_kb":-39659837,"num_write":-420889,"num_write_kb":-154900235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001850"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.308},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:22.297 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:22.571 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:22.572 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:22.584 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1453,"stamp":"2023-12-16T11:45:22.236906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-251658294,"num_objects":-95,"num_object_clones":0,"num_object_copies":-190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-95,"num_whiteouts":0,"num_read":-110288,"num_read_kb":-39659837,"num_write":-420889,"num_write_kb":-154900235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001850"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.308},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:22.585 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:22.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:22.585 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:22.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:22.585 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:22.585 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:22.586 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:22.586 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:22.586 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:22.586 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:22.586 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:22.586 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 528, 'reported_epoch': 379, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:14.291942+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:14.291942+0000', 'last_peered': '2023-12-16T11:45:14.291942+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:14.291942+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:22.586 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:22.926 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:22.926 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2160,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:22.939 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:22.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:23.215 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:23.215 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:23.228 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1453,"stamp":"2023-12-16T11:45:22.236906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-251658294,"num_objects":-95,"num_object_clones":0,"num_object_copies":-190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-95,"num_whiteouts":0,"num_read":-110288,"num_read_kb":-39659837,"num_write":-420889,"num_write_kb":-154900235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001850"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.308},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738803,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:25.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-16T11:45:26.230 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:26.517 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:26.518 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:26.529 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1455,"stamp":"2023-12-16T11:45:26.237758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-1960,"num_read_kb":-1874,"num_write":-369,"num_write_kb":-300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002056"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.308},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738804,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.439},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:26.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:26.804 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:26.805 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:26.817 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1455,"stamp":"2023-12-16T11:45:26.237758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-1960,"num_read_kb":-1874,"num_write":-369,"num_write_kb":-300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002056"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.308},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738804,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.439},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:26.818 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:26.818 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:26.818 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:26.818 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:26.818 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:26.818 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:26.819 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:26.819 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:26.819 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:26.819 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:26.819 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:26.819 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 528, 'reported_epoch': 379, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:14.291942+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:14.291942+0000', 'last_peered': '2023-12-16T11:45:14.291942+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:14.291942+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:26.819 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:27.159 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:27.159 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2164,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:27.172 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:27.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:27.449 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:27.449 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:27.462 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1455,"stamp":"2023-12-16T11:45:26.237758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-1960,"num_read_kb":-1874,"num_write":-369,"num_write_kb":-300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002056"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.308},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.32200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738804,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.439},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:30.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:30.752 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:30.752 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:30.765 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1457,"stamp":"2023-12-16T11:45:30.238555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002430"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.27400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.245},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.30199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:30.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:30.953 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-16T11:45:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:31.041 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:31.054 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1457,"stamp":"2023-12-16T11:45:30.238555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002430"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.27400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.245},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.30199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:31.055 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:31.056 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:31.056 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:31.056 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:31.056 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 528, 'reported_epoch': 379, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:14.291942+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:14.291942+0000', 'last_peered': '2023-12-16T11:45:14.291942+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:14.291942+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:31.056 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:31.399 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:31.399 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2168,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:31.412 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:31.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:31.688 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:31.700 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1457,"stamp":"2023-12-16T11:45:30.238555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002430"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.27400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.245},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.30199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:34.702 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:34.990 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:34.991 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:35.003 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1459,"stamp":"2023-12-16T11:45:34.239235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002328"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.434}]}]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.27400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:35.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:35.280 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:35.280 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:35.292 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1459,"stamp":"2023-12-16T11:45:34.239235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002328"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.434}]}]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.27400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:35.293 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:35.294 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:35.295 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:35.295 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:35.295 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 528, 'reported_epoch': 379, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:14.291942+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:14.291942+0000', 'last_peered': '2023-12-16T11:45:14.291942+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:14.291942+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:35.295 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:35.635 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:35.635 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2173,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:35.648 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:35.648 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:35.924 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:35.925 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:35.937 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1459,"stamp":"2023-12-16T11:45:34.239235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002328"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.434}]}]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:44:35 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.4410000000000001,"15min":0.72499999999999998},"min":{"1min":0.20699999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.4480000000000004,"5min":74.266000000000005,"15min":74.266000000000005},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3620000000000001,"15min":0.67700000000000005},"min":{"1min":0.23899999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":11.497,"5min":73.727000000000004,"15min":73.727000000000004},"last":0.27400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:36.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-16T11:45:38.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:39.226 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:39.227 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:39.239 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1461,"stamp":"2023-12-16T11:45:38.240366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:39.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:39.515 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:39.527 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1461,"stamp":"2023-12-16T11:45:38.240366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:39.529 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:39.529 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:39.529 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:39.529 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:39.530 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:39.530 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:39.530 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:39.530 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:39.530 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:39.531 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:39.531 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:39.531 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 528, 'reported_epoch': 379, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:14.291942+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:14.291942+0000', 'last_peered': '2023-12-16T11:45:14.291942+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:14.291942+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:39.531 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:39.871 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:39.871 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2177,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:39.884 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:39.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:40.161 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:40.162 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:40.174 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1461,"stamp":"2023-12-16T11:45:38.240366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738807,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:41.761 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-16T11:45:43.176 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:43.464 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:43.464 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:43.476 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1463,"stamp":"2023-12-16T11:45:42.241400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.216},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:43.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:43.756 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:43.756 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:43.768 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1463,"stamp":"2023-12-16T11:45:42.241400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":528,"reported_epoch":379,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:14.291942+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:14.291942+0000","last_peered":"2023-12-16T11:45:14.291942+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:14.291942+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.216},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738807,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:43.769 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:43.770 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:43.770 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:43.770 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:43.770 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 528, 'reported_epoch': 379, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:14.291942+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:14.291942+0000', 'last_peered': '2023-12-16T11:45:14.291942+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:14.291942+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:43.770 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:44.108 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:44.109 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2181,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:44.122 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:44.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:44.396 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:44.397 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:44.410 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1464,"stamp":"2023-12-16T11:45:44.241906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003028"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.216},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:47.165 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-16T11:45:47.412 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:47.701 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:47.701 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:47.713 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2023-12-16T11:45:46.242370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.216},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:47.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:47.989 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:47.990 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:48.002 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2023-12-16T11:45:46.242370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003134"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.024775+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:15.024775+0000","last_peered":"2023-12-16T11:45:15.024775+0000","last_clean":"2023-12-16T11:45:15.024775+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:15.024775+0000","last_undegraded":"2023-12-16T11:45:15.024775+0000","last_fullsized":"2023-12-16T11:45:15.024775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:44:43 2023","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.39200000000000002,"15min":0.38400000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.41899999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.216},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.54300000000000004,"15min":0.42299999999999999},"min":{"1min":0.21199999999999999,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.95999999999999996,"5min":5.391,"15min":5.391},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.34399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:48.003 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:48.004 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:48.004 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:48.004 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:48.004 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:48.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:48.346 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:48.346 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2185,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:48.359 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:48.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:48.634 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:48.635 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:48.648 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1466,"stamp":"2023-12-16T11:45:48.242725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002904"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.43099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.36799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:51.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:51.939 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:51.939 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:51.951 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1467,"stamp":"2023-12-16T11:45:50.243148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002782"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.43099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.36799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:51.952 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:52.230 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:52.231 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:52.243 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1467,"stamp":"2023-12-16T11:45:50.243148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002782"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738809,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.43099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.36799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:52.244 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:52.244 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:52.244 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:52.245 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:52.246 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:52.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-16T11:45:52.584 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:52.584 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2190,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:52.597 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:52.597 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:52.875 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:52.875 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:52.887 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1468,"stamp":"2023-12-16T11:45:52.243495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002504"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.30199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738809,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.36799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:55.888 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:56.178 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:56.178 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:56.190 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1469,"stamp":"2023-12-16T11:45:54.244042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002641"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.30199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.214}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:56.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:56.464 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:56.464 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:56.476 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1470,"stamp":"2023-12-16T11:45:56.244666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002759"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.30199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.214}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:56.477 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:45:56.477 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:56.477 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:45:56.477 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:45:56.478 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:45:56.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:45:56.816 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:56.817 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2194,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:45:56.829 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:45:56.830 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:45:57.104 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:45:57.104 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:45:57.116 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1470,"stamp":"2023-12-16T11:45:56.244666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002759"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.30199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.214}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:45:57.973 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-16T11:46:00.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:00.407 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:00.407 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:00.420 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1472,"stamp":"2023-12-16T11:46:00.245546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002821"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.214},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.28100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:00.421 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:00.695 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:00.695 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:00.707 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1472,"stamp":"2023-12-16T11:46:00.245546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002821"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.214},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.28100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:00.708 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:00.709 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:00.709 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:00.709 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:00.709 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:00.709 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:01.045 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:01.045 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2198,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:01.058 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:01.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:01.336 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:01.349 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1472,"stamp":"2023-12-16T11:46:00.245546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002821"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738811,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.214},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.28100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:03.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-16T11:46:04.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:04.637 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:04.637 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:04.649 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1474,"stamp":"2023-12-16T11:46:04.246220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.214},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738812,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.221},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:04.650 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:04.927 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:04.927 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:04.939 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1474,"stamp":"2023-12-16T11:46:04.246220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.214},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738812,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.221},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:04.940 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:04.940 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:04.941 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:04.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:05.285 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:05.286 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2202,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:05.299 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:05.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:05.576 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:05.576 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:05.588 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1474,"stamp":"2023-12-16T11:46:04.246220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.214},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738812,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.221},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:08.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:08.781 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-16T11:46:08.876 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:08.888 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1476,"stamp":"2023-12-16T11:46:08.247180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.223}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:08.889 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:09.163 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:09.163 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:09.175 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1476,"stamp":"2023-12-16T11:46:08.247180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.223}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:09.176 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:09.176 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:09.176 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:09.176 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:09.177 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:09.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:09.519 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:09.519 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2207,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:09.532 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:09.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:09.808 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:09.809 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:09.821 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1476,"stamp":"2023-12-16T11:46:08.247180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.223}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:12.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:13.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:13.109 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:13.121 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1478,"stamp":"2023-12-16T11:46:12.247944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002397"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:13.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:13.398 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:13.398 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:13.410 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1478,"stamp":"2023-12-16T11:46:12.247944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002397"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:13.411 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:13.411 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:13.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:13.413 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:13.752 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:13.752 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2211,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:13.765 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:13.765 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:14.041 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:14.053 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1478,"stamp":"2023-12-16T11:46:12.247944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002397"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:14.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-16T11:46:17.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:17.344 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:17.345 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:17.356 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1480,"stamp":"2023-12-16T11:46:16.248827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.432},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.47999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:17.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:17.632 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:17.632 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:17.644 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1480,"stamp":"2023-12-16T11:46:16.248827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.21199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.432},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.47999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:17.645 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:17.645 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:17.646 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:17.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:17.986 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:17.986 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2215,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:17.999 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:18.000 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:18.278 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:18.278 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:18.290 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1481,"stamp":"2023-12-16T11:46:18.249212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.40500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:19.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-16T11:46:21.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:21.581 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:21.594 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1482,"stamp":"2023-12-16T11:46:20.249622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.40500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:21.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:21.973 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:21.973 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:21.985 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1482,"stamp":"2023-12-16T11:46:20.249622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002441"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.40500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:21.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:21.986 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:21.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:21.986 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:21.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:21.986 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:21.987 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:21.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:21.987 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:21.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:21.987 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:21.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:21.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:22.327 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:22.327 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2219,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:22.340 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:22.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:22.614 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:22.614 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:22.626 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1483,"stamp":"2023-12-16T11:46:22.250203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002621"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.222},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.40500000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:24.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-16T11:46:25.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:25.915 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:25.916 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:25.928 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2023-12-16T11:46:24.250614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.222},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:25.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:26.204 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:26.204 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:26.217 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2023-12-16T11:46:24.250614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.222},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296951+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:14.296951+0000', 'last_peered': '2023-12-16T11:45:14.296951+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:14.296951+0000', 'last_undegraded': '2023-12-16T11:45:14.296951+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:26.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:26.219 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:26.219 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:26.219 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:26.219 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:26.219 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:26.558 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:26.558 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2224,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:26.571 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:26.572 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:26.846 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:26.847 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:26.859 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1485,"stamp":"2023-12-16T11:46:26.251032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002576"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296951+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:14.296951+0000","last_peered":"2023-12-16T11:45:14.296951+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:14.296951+0000","last_undegraded":"2023-12-16T11:45:14.296951+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.222},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:29.860 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:30.149 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:30.150 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:30.162 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1486,"stamp":"2023-12-16T11:46:28.251575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002747"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:30.163 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:30.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-16T11:46:30.438 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:30.439 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:30.451 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1487,"stamp":"2023-12-16T11:46:30.252203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002990"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 397, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.299442+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:14.299442+0000', 'last_peered': '2023-12-16T11:45:14.299442+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:14.299442+0000', 'last_undegraded': '2023-12-16T11:45:14.299442+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:30.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:30.453 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:30.453 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:30.453 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:30.453 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:30.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:30.794 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:30.794 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2228,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:30.807 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:30.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:31.082 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:31.094 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1487,"stamp":"2023-12-16T11:46:30.252203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002990"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":397,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.299442+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:14.299442+0000","last_peered":"2023-12-16T11:45:14.299442+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:14.299442+0000","last_undegraded":"2023-12-16T11:45:14.299442+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.23999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.22600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:34.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:34.386 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:34.386 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:34.398 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1489,"stamp":"2023-12-16T11:46:34.252961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002757"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:34.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:34.675 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:34.675 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:34.687 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1489,"stamp":"2023-12-16T11:46:34.252961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002757"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:34.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:34.689 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:34.690 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:34.690 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:34.690 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:34.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:35.029 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:35.029 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2232,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:35.042 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:35.042 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:35.320 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:35.320 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:35.332 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1489,"stamp":"2023-12-16T11:46:34.252961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002757"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:35.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-16T11:46:38.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:38.623 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:38.635 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1491,"stamp":"2023-12-16T11:46:38.253871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002838"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:38.636 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:38.909 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:38.909 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:38.921 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1491,"stamp":"2023-12-16T11:46:38.253871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002838"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:38.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:38.923 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:39.267 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:39.267 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2236,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:39.280 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:39.280 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:39.560 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:39.572 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1491,"stamp":"2023-12-16T11:46:38.253871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002838"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.22900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:41.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-16T11:46:42.574 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:42.859 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:42.860 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:42.873 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1493,"stamp":"2023-12-16T11:46:42.254823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002619"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.377}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:42.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:43.147 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:43.147 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:43.159 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1493,"stamp":"2023-12-16T11:46:42.254823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002619"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.377}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:43.160 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:43.161 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:43.161 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.298224+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:14.298224+0000', 'last_peered': '2023-12-16T11:45:14.298224+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:14.298224+0000', 'last_undegraded': '2023-12-16T11:45:14.298224+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:43.161 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:43.161 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:43.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:43.500 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:43.501 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2241,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:43.513 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:43.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:43.788 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:43.788 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:43.800 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1493,"stamp":"2023-12-16T11:46:42.254823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002619"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.298224+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:14.298224+0000","last_peered":"2023-12-16T11:45:14.298224+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:14.298224+0000","last_undegraded":"2023-12-16T11:45:14.298224+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.377}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.58699999999999997,"15min":0.72399999999999998},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.45200000000000001,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.57599999999999996,"15min":0.67900000000000005},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.499,"5min":11.497,"15min":73.727000000000004},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:46.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-16T11:46:46.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:47.088 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:47.089 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:47.101 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1495,"stamp":"2023-12-16T11:46:46.255741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002780"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.377}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.435},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:47.102 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:47.378 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:47.379 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:47.391 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1495,"stamp":"2023-12-16T11:46:46.255741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002780"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.377}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.435},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:47.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:47.393 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:47.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:47.732 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:47.732 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2245,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:47.746 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:47.746 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:48.020 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:48.020 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:48.032 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1495,"stamp":"2023-12-16T11:46:46.255741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002780"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.377}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.435},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.51000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:51.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:51.319 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:51.320 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:51.334 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1497,"stamp":"2023-12-16T11:46:50.256528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002657"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.184},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.16800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.216},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:51.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:51.611 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:51.611 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:51.624 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1497,"stamp":"2023-12-16T11:46:50.256528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002657"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.184},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.16800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.216},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:51.624 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:51.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:51.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:51.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:51.964 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:51.964 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2249,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:51.977 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:51.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:52.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-16T11:46:52.254 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:52.254 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:52.266 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1497,"stamp":"2023-12-16T11:46:50.256528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002657"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:45:45 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.216,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.47199999999999998,"5min":3.786,"15min":5.2119999999999997},"last":0.184},{"interface":"front","average":{"1min":0.308,"5min":0.52400000000000002,"15min":0.41799999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.17999999999999999},"max":{"1min":0.53100000000000003,"5min":5.391,"15min":5.391},"last":0.16800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.216},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:55.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:55.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:55.556 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:55.568 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1499,"stamp":"2023-12-16T11:46:54.257257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002434"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.251},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.432}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:55.569 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:55.845 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:55.845 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:55.857 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1499,"stamp":"2023-12-16T11:46:54.257257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002434"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.251},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.432}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:46:55.858 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:55.859 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:46:55.859 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:55.859 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:46:55.859 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:46:55.859 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:46:56.197 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:56.198 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2253,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:46:56.210 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:46:56.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:56.488 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:56.488 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:56.500 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1500,"stamp":"2023-12-16T11:46:56.257741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002470"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.251},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.27500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.432}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:57.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2023-12-16T11:46:59.501 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:46:59.793 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:46:59.793 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:46:59.805 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2023-12-16T11:46:58.258178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002436"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:46:59.806 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:00.081 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:00.082 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:00.094 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2023-12-16T11:46:58.258178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002436"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:00.095 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:00.096 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:00.096 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:00.096 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:00.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:00.439 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:00.439 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2258,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:00.452 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:00.453 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:00.728 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:00.729 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:00.741 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1502,"stamp":"2023-12-16T11:47:00.258696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:02.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-16T11:47:03.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:04.029 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:04.030 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:04.042 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1503,"stamp":"2023-12-16T11:47:02.259200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002671"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:04.043 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:04.320 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:04.320 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:04.333 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1504,"stamp":"2023-12-16T11:47:04.259724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:04.334 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:04.335 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:04.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:04.672 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:04.672 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2262,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:04.685 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:04.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:04.961 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:04.962 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:04.974 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1504,"stamp":"2023-12-16T11:47:04.259724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:07.976 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:08.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-16T11:47:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:08.262 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:08.274 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1505,"stamp":"2023-12-16T11:47:06.260276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:15.023818+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:15.023818+0000","last_peered":"2023-12-16T11:45:15.023818+0000","last_clean":"2023-12-16T11:45:15.023818+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:15.023818+0000","last_undegraded":"2023-12-16T11:45:15.023818+0000","last_fullsized":"2023-12-16T11:45:15.023818+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:08.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:08.550 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:08.550 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:08.562 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1506,"stamp":"2023-12-16T11:47:08.260666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002925"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.28299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:08.563 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:08.563 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:08.563 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:08.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:08.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:08.904 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2266,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:08.917 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:08.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:09.191 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:09.192 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:09.204 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1506,"stamp":"2023-12-16T11:47:08.260666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002925"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.28299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:12.206 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:12.495 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:12.496 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:12.508 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1508,"stamp":"2023-12-16T11:47:12.261466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:12.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:12.785 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:12.785 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:12.797 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1508,"stamp":"2023-12-16T11:47:12.261466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:12.798 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:12.798 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:12.798 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:12.798 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.293311+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:14.293311+0000', 'last_peered': '2023-12-16T11:45:14.293311+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:14.293311+0000', 'last_undegraded': '2023-12-16T11:45:14.293311+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:12.799 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:12.799 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:13.137 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:13.137 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2270,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:13.150 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:13.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:13.425 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:13.426 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:13.438 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1508,"stamp":"2023-12-16T11:47:12.261466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.293311+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:14.293311+0000","last_peered":"2023-12-16T11:45:14.293311+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:14.293311+0000","last_undegraded":"2023-12-16T11:45:14.293311+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":394,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-12-16T11:45:14.295452+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:14.295452+0000","last_peered":"2023-12-16T11:45:14.295452+0000","last_clean":"2023-12-16T11:45:14.295452+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:14.295452+0000","last_undegraded":"2023-12-16T11:45:14.295452+0000","last_fullsized":"2023-12-16T11:45:14.295452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:13.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-16T11:47:16.439 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:16.729 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:16.741 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1510,"stamp":"2023-12-16T11:47:16.262193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002469"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:16.742 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:17.016 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:17.016 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:17.029 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1510,"stamp":"2023-12-16T11:47:16.262193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002469"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:17.029 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 379, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:14.296507+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:14.296507+0000', 'last_peered': '2023-12-16T11:45:14.296507+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:14.296507+0000', 'last_undegraded': '2023-12-16T11:45:14.296507+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:17.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:17.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:17.373 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:17.373 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2274,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:17.386 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:17.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:17.662 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:17.663 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:17.675 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1510,"stamp":"2023-12-16T11:47:16.262193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002469"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":398,"reported_epoch":379,"state":"active+undersized","last_fresh":"2023-12-16T11:45:14.296507+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:14.296507+0000","last_peered":"2023-12-16T11:45:14.296507+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:14.296507+0000","last_undegraded":"2023-12-16T11:45:14.296507+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:19.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-16T11:47:20.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:20.964 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:20.965 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:20.977 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1512,"stamp":"2023-12-16T11:47:20.263144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:20.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:21.253 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:21.253 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:21.265 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1512,"stamp":"2023-12-16T11:47:20.263144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:21.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:21.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:21.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:21.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:21.267 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:21.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:21.605 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2279,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:21.618 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:21.618 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:21.895 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:21.895 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:21.908 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1512,"stamp":"2023-12-16T11:47:20.263144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.29999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:24.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-16T11:47:24.910 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:25.199 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:25.200 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:25.212 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1514,"stamp":"2023-12-16T11:47:24.264025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002559"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:25.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:25.489 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:25.490 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:25.502 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1514,"stamp":"2023-12-16T11:47:24.264025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002559"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:25.503 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:25.504 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:25.504 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:25.504 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:25.842 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:25.842 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2283,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:25.855 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:25.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:26.130 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:26.142 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1514,"stamp":"2023-12-16T11:47:24.264025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002559"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.27200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:29.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:29.433 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:29.434 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:29.445 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1516,"stamp":"2023-12-16T11:47:28.265082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:29.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:29.719 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:29.720 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:29.732 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1516,"stamp":"2023-12-16T11:47:28.265082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:29.732 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:29.733 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:29.734 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:29.734 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:29.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:29.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-16T11:47:30.073 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:30.073 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2287,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:30.086 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:30.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:30.363 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:30.364 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:30.376 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1517,"stamp":"2023-12-16T11:47:30.265561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.21099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:33.377 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:33.730 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:33.731 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:33.743 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1518,"stamp":"2023-12-16T11:47:32.265948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.219}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:33.744 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:34.018 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:34.019 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:34.031 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1518,"stamp":"2023-12-16T11:47:32.265948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.219}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:34.032 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:34.033 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:34.033 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:34.033 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:34.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:34.371 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:34.371 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2291,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:34.386 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:34.386 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:34.662 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:34.663 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:34.675 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1519,"stamp":"2023-12-16T11:47:34.266374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002711"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.219}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:35.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-16T11:47:37.678 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:37.965 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:37.966 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:37.978 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1520,"stamp":"2023-12-16T11:47:36.266761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002736"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.219}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:37.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:38.252 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:38.252 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:38.265 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1520,"stamp":"2023-12-16T11:47:36.266761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002736"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.219}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:38.266 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:38.267 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:38.267 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:38.267 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:38.608 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2296,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:38.621 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:38.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:38.899 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:38.899 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:38.912 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1521,"stamp":"2023-12-16T11:47:38.267179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002624"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.41799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:40.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-16T11:47:41.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:42.203 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:42.203 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:42.215 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1522,"stamp":"2023-12-16T11:47:40.267781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002698"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.41799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:42.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:42.495 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:42.495 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:42.508 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1523,"stamp":"2023-12-16T11:47:42.268310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.312},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:42.509 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:42.509 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:42.509 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:42.509 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:42.509 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:42.509 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:42.510 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:42.510 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:42.510 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:42.510 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:42.510 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:42.510 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 529, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:16.568158+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:16.568158+0000', 'last_peered': '2023-12-16T11:45:16.568158+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:16.568158+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:42.511 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:42.848 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:42.849 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2300,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:42.862 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:42.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:43.138 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:43.138 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:43.150 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1523,"stamp":"2023-12-16T11:47:42.268310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":529,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:16.568158+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:16.568158+0000","last_peered":"2023-12-16T11:45:16.568158+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:16.568158+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.312},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:46:38 2023","interfaces":[{"interface":"back","average":{"1min":0.253,"5min":0.501,"15min":0.71999999999999997},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.158},"max":{"1min":0.432,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.56399999999999995,"15min":0.67700000000000005},"min":{"1min":0.214,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47999999999999998,"5min":11.497,"15min":73.727000000000004},"last":0.33400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:46.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-16T11:47:46.152 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:46.441 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:46.441 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:46.453 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1525,"stamp":"2023-12-16T11:47:46.269172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002798"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.312},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:46.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:46.732 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:46.732 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:46.745 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1525,"stamp":"2023-12-16T11:47:46.269172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002798"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.312},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:46.748 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:46.748 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:46.748 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:46.749 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:46.750 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:47.092 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:47.092 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2304,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:47.105 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:47.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:47.384 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:47.384 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:47.397 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1525,"stamp":"2023-12-16T11:47:46.269172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002798"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.295620+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:16.295620+0000","last_peered":"2023-12-16T11:45:16.295620+0000","last_clean":"2023-12-16T11:45:16.295620+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:16.295620+0000","last_undegraded":"2023-12-16T11:45:16.295620+0000","last_fullsized":"2023-12-16T11:45:16.295620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.312},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.33600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:50.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:50.683 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:50.683 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:50.695 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1527,"stamp":"2023-12-16T11:47:50.269947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.219},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.23999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:50.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:50.972 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:50.973 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:50.985 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1527,"stamp":"2023-12-16T11:47:50.269947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.219},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.23999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:50.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:50.986 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:50.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:50.986 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:50.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:50.986 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:50.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:50.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:50.987 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:50.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:50.987 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:50.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:50.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:51.324 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:51.324 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2308,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:51.337 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:51.337 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:51.461 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-16T11:47:51.613 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:51.613 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:51.625 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1527,"stamp":"2023-12-16T11:47:50.269947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:46:49 2023","interfaces":[{"interface":"back","average":{"1min":0.23999999999999999,"5min":0.33500000000000002,"15min":0.38100000000000001},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.32700000000000001,"5min":3.786,"15min":5.2119999999999997},"last":0.219},{"interface":"front","average":{"1min":0.251,"5min":0.45400000000000001,"15min":0.41499999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.377,"5min":5.391,"15min":5.391},"last":0.23999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:54.628 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:54.916 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:54.916 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:54.928 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1529,"stamp":"2023-12-16T11:47:54.270888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:54.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:55.203 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:55.204 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:55.216 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1529,"stamp":"2023-12-16T11:47:54.270888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:55.217 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:55.217 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:55.217 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:55.217 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:55.217 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:55.217 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:55.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:55.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:55.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:55.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:55.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:55.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:55.218 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:55.558 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:55.558 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2313,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:55.571 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:55.571 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:55.847 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:55.848 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:55.860 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1529,"stamp":"2023-12-16T11:47:54.270888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002577"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.29299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:56.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-16T11:47:58.862 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:59.148 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:59.148 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:59.160 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2023-12-16T11:47:58.271783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002610"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.443},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.41299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:59.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:47:59.437 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:59.438 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:47:59.450 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2023-12-16T11:47:58.271783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002610"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.443},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.41299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:47:59.451 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:47:59.451 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:59.451 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:47:59.451 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:47:59.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:47:59.452 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:47:59.791 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:47:59.792 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2317,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:47:59.804 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:47:59.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:00.081 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:00.082 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:00.094 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2023-12-16T11:47:58.271783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002610"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.443},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.41299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:02.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-16T11:48:03.095 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:03.386 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:03.387 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:03.398 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1533,"stamp":"2023-12-16T11:48:02.272677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002730"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:03.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:03.674 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:03.675 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:03.687 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1533,"stamp":"2023-12-16T11:48:02.272677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002730"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:03.688 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:03.689 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:03.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:04.032 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:04.049 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2321,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:04.050 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:04.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:04.327 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:04.328 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:04.340 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1534,"stamp":"2023-12-16T11:48:04.273066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002676"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:07.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:07.628 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:07.628 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:07.640 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1535,"stamp":"2023-12-16T11:48:06.273615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:07.641 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:07.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-16T11:48:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:07.918 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:07.930 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1535,"stamp":"2023-12-16T11:48:06.273615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.24199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:07.931 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:07.932 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:07.932 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:07.932 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:07.932 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:07.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:08.273 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2325,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:08.286 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:08.286 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:08.560 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:08.560 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:08.572 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1536,"stamp":"2023-12-16T11:48:08.274008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:11.573 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:11.861 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:11.862 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:11.874 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1537,"stamp":"2023-12-16T11:48:10.274562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:11.875 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:12.150 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:12.150 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:12.162 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1537,"stamp":"2023-12-16T11:48:10.274562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.251}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:12.163 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:12.163 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:12.163 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:12.163 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:12.163 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:12.163 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:12.164 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:12.164 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:12.164 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:12.164 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:12.164 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:12.164 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:12.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:12.506 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:12.506 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2330,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:12.519 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:12.519 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:12.793 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:12.794 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:12.806 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1538,"stamp":"2023-12-16T11:48:12.274938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:13.077 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-16T11:48:15.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:16.093 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:16.093 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:16.105 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1539,"stamp":"2023-12-16T11:48:14.275403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:16.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:16.382 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:16.383 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:16.395 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1540,"stamp":"2023-12-16T11:48:16.275963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002896"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:16.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:16.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:16.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:16.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:16.396 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:16.396 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:16.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:16.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:16.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:16.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:16.397 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:16.397 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:16.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:16.740 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2334,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:16.753 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:16.753 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:17.027 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:17.027 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:17.039 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1540,"stamp":"2023-12-16T11:48:16.275963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002896"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.375}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:18.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-12-16T11:48:20.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:20.327 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:20.328 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:20.339 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1542,"stamp":"2023-12-16T11:48:20.276874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002865"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.253},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:20.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:20.615 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:20.616 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:20.628 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1542,"stamp":"2023-12-16T11:48:20.276874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002865"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.253},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:20.629 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:20.630 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:20.630 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:20.630 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:20.630 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:20.630 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:20.973 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:20.973 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2338,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:20.986 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:20.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:21.261 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:21.262 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:21.273 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1542,"stamp":"2023-12-16T11:48:20.276874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002865"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.253},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.26600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.40699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:23.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-16T11:48:24.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:24.562 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:24.563 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:24.575 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2023-12-16T11:48:24.277826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.192},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.23499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:24.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:24.851 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:24.863 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2023-12-16T11:48:24.277826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.192},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.23499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:24.864 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:24.864 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:24.864 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:24.864 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:24.864 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:24.864 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:24.865 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:24.865 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:24.865 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:24.865 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:24.865 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:24.865 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:24.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:25.203 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:25.204 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2342,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:25.216 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:25.216 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:25.491 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:25.492 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:25.503 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2023-12-16T11:48:24.277826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.192},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.23499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:28.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:28.791 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:28.792 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:28.803 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1546,"stamp":"2023-12-16T11:48:28.278522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002559"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:28.804 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:29.078 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:29.078 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:29.090 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1546,"stamp":"2023-12-16T11:48:28.278522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002559"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:29.091 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:29.091 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:29.091 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:29.091 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:29.092 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:29.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:29.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-16T11:48:29.436 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:29.437 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2347,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:29.450 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:29.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:29.724 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:29.725 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:29.737 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1546,"stamp":"2023-12-16T11:48:28.278522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002559"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:32.739 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:33.028 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:33.028 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:33.040 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1548,"stamp":"2023-12-16T11:48:32.279492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002618"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:33.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:33.315 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:33.315 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:33.327 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1548,"stamp":"2023-12-16T11:48:32.279492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002618"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:33.328 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:33.328 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:33.328 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:33.328 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:33.329 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:33.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:33.673 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:33.673 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2351,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:33.686 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:33.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:33.961 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:33.961 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:33.973 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1548,"stamp":"2023-12-16T11:48:32.279492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002618"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.32800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:34.693 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-12-16T11:48:36.975 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:37.263 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:37.264 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:37.276 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1550,"stamp":"2023-12-16T11:48:36.280416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002589"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:37.277 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:37.552 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:37.553 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:37.565 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1550,"stamp":"2023-12-16T11:48:36.280416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002589"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570706+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:45:16.570706+0000', 'last_peered': '2023-12-16T11:45:16.570706+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:45:16.570706+0000', 'last_undegraded': '2023-12-16T11:45:16.570706+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:37.566 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:37.567 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:37.567 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:37.567 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:37.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:37.909 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:37.909 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2355,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:37.922 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:37.922 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:38.197 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:38.198 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:38.210 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1550,"stamp":"2023-12-16T11:48:36.280416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002589"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570706+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:45:16.570706+0000","last_peered":"2023-12-16T11:45:16.570706+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:45:16.570706+0000","last_undegraded":"2023-12-16T11:45:16.570706+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.27200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:40.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-16T11:48:41.211 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:41.499 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:41.499 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:41.512 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1552,"stamp":"2023-12-16T11:48:40.281414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002891"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:41.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:41.787 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:41.788 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:41.800 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1552,"stamp":"2023-12-16T11:48:40.281414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002891"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.25}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:41.801 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:41.801 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 398, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.570811+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:45:16.570811+0000', 'last_peered': '2023-12-16T11:45:16.570811+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:45:16.570811+0000', 'last_undegraded': '2023-12-16T11:45:16.570811+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:41.801 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:41.801 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:41.801 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:41.801 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:41.801 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:41.802 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:41.802 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:41.802 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:41.802 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:41.802 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:41.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:42.144 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:42.144 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2359,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:42.157 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:42.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:42.433 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:42.434 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:42.445 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1553,"stamp":"2023-12-16T11:48:42.281802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002757"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":398,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.570811+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:45:16.570811+0000","last_peered":"2023-12-16T11:45:16.570811+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:45:16.570811+0000","last_undegraded":"2023-12-16T11:45:16.570811+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.221},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.23799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.222},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:45.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:45.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-12-16T11:48:45.733 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:45.733 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:45.746 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2023-12-16T11:48:44.282186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.221},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.23799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.215},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:45.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:46.023 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:46.023 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:46.035 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2023-12-16T11:48:44.282186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002693"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.221},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.23799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.215},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:46.036 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:46.036 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:46.036 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:46.036 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 12, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.569231+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:45:16.569231+0000', 'last_peered': '2023-12-16T11:45:16.569231+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:45:16.569231+0000', 'last_undegraded': '2023-12-16T11:45:16.569231+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:46.037 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:46.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:46.378 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2363,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:46.391 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:46.392 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:46.870 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:46.870 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:46.882 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1555,"stamp":"2023-12-16T11:48:46.282820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.569231+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:45:16.569231+0000","last_peered":"2023-12-16T11:45:16.569231+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:45:16.569231+0000","last_undegraded":"2023-12-16T11:45:16.569231+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.221},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.23799999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:47:40 2023","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":0.48999999999999999,"15min":0.71999999999999997},"min":{"1min":0.19500000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.435,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.215},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.55500000000000005,"15min":0.68000000000000005},"min":{"1min":0.251,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.51000000000000001,"5min":11.497,"15min":73.727000000000004},"last":0.26000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:49.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:50.170 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:50.171 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:50.183 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1556,"stamp":"2023-12-16T11:48:48.283327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002910"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.40999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:50.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:50.461 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:50.462 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:50.473 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1557,"stamp":"2023-12-16T11:48:50.283884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.40999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:50.474 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:50.474 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:50.474 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:50.474 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:50.475 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:50.475 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:50.815 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:50.815 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2368,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:50.828 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:50.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:50.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-12-16T11:48:51.103 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:51.104 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:51.116 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1557,"stamp":"2023-12-16T11:48:50.283884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:47:49 2023","interfaces":[{"interface":"back","average":{"1min":0.27000000000000002,"5min":0.27800000000000002,"15min":0.38},"min":{"1min":0.19800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39800000000000002,"5min":0.93400000000000005,"15min":5.2119999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.28499999999999998,"5min":0.33400000000000002,"15min":0.41499999999999998},"min":{"1min":0.21099999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.41799999999999998,"5min":2.3130000000000002,"15min":5.391},"last":0.40999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.27300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:54.118 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:54.408 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:54.408 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:54.421 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1559,"stamp":"2023-12-16T11:48:54.284694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002892"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.222},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.23599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.216},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.24199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:54.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:54.700 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:54.701 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:54.713 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1559,"stamp":"2023-12-16T11:48:54.284694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002892"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.222},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.23599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.216},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.24199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:54.714 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:54.715 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:54.715 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:54.715 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:55.054 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:55.054 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2372,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:55.067 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:55.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:55.344 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:55.345 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:55.356 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1559,"stamp":"2023-12-16T11:48:54.284694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002892"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.222},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.23599999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.216},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.24199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:56.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-16T11:48:58.358 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:58.652 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:58.653 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:58.665 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1561,"stamp":"2023-12-16T11:48:58.285516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.251},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.22500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:58.666 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:58.944 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:58.944 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:58.957 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1561,"stamp":"2023-12-16T11:48:58.285516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.251},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.22500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:48:58.958 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:58.959 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:48:58.959 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:48:58.959 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:48:59.301 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:59.301 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2376,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:48:59.314 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:48:59.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:48:59.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:48:59.591 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:48:59.605 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1561,"stamp":"2023-12-16T11:48:58.285516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.251},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.22500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:01.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-12-16T11:49:02.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:02.893 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:02.893 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:02.905 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2023-12-16T11:49:02.286412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.432},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:02.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:03.181 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:03.182 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:03.194 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2023-12-16T11:49:02.286412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.432},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:03.195 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:03.196 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:03.196 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:03.196 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:03.196 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:03.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:03.542 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:03.542 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2381,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:03.555 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:03.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:03.829 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:03.829 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:03.841 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2023-12-16T11:49:02.286412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002527"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.432},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:06.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:07.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-12-16T11:49:07.129 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:07.130 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:07.142 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-12-16T11:49:06.287146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.432},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:07.143 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:07.418 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:07.418 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:07.430 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-12-16T11:49:06.287146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.432},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:07.431 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:07.431 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:07.431 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:07.431 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:07.431 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:07.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:07.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:07.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:07.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:07.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:07.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:07.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:07.432 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:07.770 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:07.770 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2385,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:07.783 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:07.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:08.058 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:08.070 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-12-16T11:49:06.287146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.432},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.46200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:11.072 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:11.360 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:11.360 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:11.372 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2023-12-16T11:49:10.288004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002488"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.35399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.185},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.224}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:11.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:11.649 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:11.662 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2023-12-16T11:49:10.288004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002488"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.294908+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:45:16.294908+0000","last_peered":"2023-12-16T11:45:16.294908+0000","last_clean":"2023-12-16T11:45:16.294908+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:45:16.294908+0000","last_undegraded":"2023-12-16T11:45:16.294908+0000","last_fullsized":"2023-12-16T11:45:16.294908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.35399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.185},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.224}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:11.663 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:11.664 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:11.664 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:11.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:12.003 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:12.003 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2389,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:12.016 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:12.016 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:12.292 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:12.293 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:12.305 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1568,"stamp":"2023-12-16T11:49:12.288402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002488"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.252},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.185},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.224}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:12.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-12-16T11:49:15.307 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:15.596 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:15.597 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:15.610 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1569,"stamp":"2023-12-16T11:49:14.288776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002364"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.252},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.248},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:15.611 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:15.888 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:15.888 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:15.900 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1569,"stamp":"2023-12-16T11:49:14.288776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002364"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.252},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.248},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:15.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:15.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:15.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:15.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:15.901 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:15.901 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:15.902 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:15.902 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.571029+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:45:16.571029+0000', 'last_peered': '2023-12-16T11:45:16.571029+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:45:16.571029+0000', 'last_undegraded': '2023-12-16T11:45:16.571029+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:15.902 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:15.902 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:15.902 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:15.902 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:15.902 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:16.239 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:16.239 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2393,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:16.252 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:16.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:16.530 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:16.531 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:16.543 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1570,"stamp":"2023-12-16T11:49:16.289281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002495"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.571029+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:45:16.571029+0000","last_peered":"2023-12-16T11:45:16.571029+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:45:16.571029+0000","last_undegraded":"2023-12-16T11:45:16.571029+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:16.572829+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:45:16.572829+0000","last_peered":"2023-12-16T11:45:16.572829+0000","last_clean":"2023-12-16T11:45:16.572829+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:45:16.572829+0000","last_undegraded":"2023-12-16T11:45:16.572829+0000","last_fullsized":"2023-12-16T11:45:16.572829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.252},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.23200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.248},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:17.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-12-16T11:49:19.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:19.834 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:19.834 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:19.849 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2023-12-16T11:49:18.289699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:19.851 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:20.126 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:20.126 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:20.138 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1571,"stamp":"2023-12-16T11:49:18.289699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:20.139 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:20.139 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:20.139 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:20.139 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:45:16.573733+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:45:16.573733+0000', 'last_peered': '2023-12-16T11:45:16.573733+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:45:16.573733+0000', 'last_undegraded': '2023-12-16T11:45:16.573733+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:12.360326+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:47:12.360326+0000', 'last_peered': '2023-12-16T11:47:12.360326+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:47:12.360326+0000', 'last_undegraded': '2023-12-16T11:47:12.360326+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:20.140 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:20.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:20.481 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:20.482 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2398,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:20.494 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:20.495 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:20.769 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:20.770 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:20.782 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1572,"stamp":"2023-12-16T11:49:20.290237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:23.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-12-16T11:49:23.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:24.070 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:24.070 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:24.082 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2023-12-16T11:49:22.290650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002645"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:45:16.573733+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:45:16.573733+0000","last_peered":"2023-12-16T11:45:16.573733+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:45:16.573733+0000","last_undegraded":"2023-12-16T11:45:16.573733+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:24.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:24.358 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:24.358 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:24.370 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-12-16T11:49:24.291062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002659"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.437}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:24.371 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:17.360859+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:47:17.360859+0000', 'last_peered': '2023-12-16T11:47:17.360859+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:47:17.360859+0000', 'last_undegraded': '2023-12-16T11:47:17.360859+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:12.360326+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:47:12.360326+0000', 'last_peered': '2023-12-16T11:47:12.360326+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:47:12.360326+0000', 'last_undegraded': '2023-12-16T11:47:12.360326+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:24.372 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:24.373 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:24.373 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:24.373 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:24.712 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2402,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:24.725 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:24.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:25.001 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:25.002 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:25.014 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-12-16T11:49:24.291062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002659"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.307}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.437}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:28.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:28.305 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:28.306 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:28.318 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-12-16T11:49:28.292144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:28.319 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:28.592 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:28.592 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:28.604 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-12-16T11:49:28.292144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:28.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:28.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:28.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:28.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:28.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:28.605 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:17.360859+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:47:17.360859+0000', 'last_peered': '2023-12-16T11:47:17.360859+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:47:17.360859+0000', 'last_undegraded': '2023-12-16T11:47:17.360859+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:28.605 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:28.606 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:12.360326+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:47:12.360326+0000', 'last_peered': '2023-12-16T11:47:12.360326+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:47:12.360326+0000', 'last_undegraded': '2023-12-16T11:47:12.360326+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:28.606 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:28.606 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:28.606 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:28.606 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:28.606 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:28.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-12-16T11:49:28.945 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:28.945 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2406,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:28.958 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:28.958 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:29.235 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:29.235 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:29.247 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-12-16T11:49:28.292144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.249}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.317}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:32.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:32.544 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:32.545 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:32.555 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-12-16T11:49:32.293003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002765"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:32.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:32.830 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:32.831 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:32.843 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-12-16T11:49:32.293003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002765"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:17.360859+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:47:17.360859+0000', 'last_peered': '2023-12-16T11:47:17.360859+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:47:17.360859+0000', 'last_undegraded': '2023-12-16T11:47:17.360859+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:12.360326+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:47:12.360326+0000', 'last_peered': '2023-12-16T11:47:12.360326+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:47:12.360326+0000', 'last_undegraded': '2023-12-16T11:47:12.360326+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:32.844 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:32.845 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:32.845 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:32.845 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:32.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:33.183 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:33.183 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2410,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:33.196 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:33.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:33.472 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:33.473 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:33.485 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-12-16T11:49:32.293003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002765"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:34.136 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-12-16T11:49:36.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:36.775 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:36.776 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:36.788 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-12-16T11:49:36.293898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002836"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:36.789 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:37.062 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:37.062 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:37.074 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-12-16T11:49:36.293898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002836"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:37.075 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-16T11:49:37.075 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:32.355808+0000', 'last_change': '2023-12-16T11:44:34.332009+0000', 'last_active': '2023-12-16T11:46:32.355808+0000', 'last_peered': '2023-12-16T11:46:32.355808+0000', 'last_clean': '2023-12-16T11:44:22.322939+0000', 'last_became_active': '2023-12-16T11:44:34.332009+0000', 'last_became_peered': '2023-12-16T11:44:34.332009+0000', 'last_unstale': '2023-12-16T11:46:32.355808+0000', 'last_undegraded': '2023-12-16T11:46:32.355808+0000', 'last_fullsized': '2023-12-16T11:44:34.319771+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:37.075 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-16T11:49:37.075 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:27.355268+0000', 'last_change': '2023-12-16T11:44:34.317288+0000', 'last_active': '2023-12-16T11:46:27.355268+0000', 'last_peered': '2023-12-16T11:46:27.355268+0000', 'last_clean': '2023-12-16T11:44:21.274929+0000', 'last_became_active': '2023-12-16T11:44:34.317288+0000', 'last_became_peered': '2023-12-16T11:44:34.317288+0000', 'last_unstale': '2023-12-16T11:46:27.355268+0000', 'last_undegraded': '2023-12-16T11:46:27.355268+0000', 'last_fullsized': '2023-12-16T11:44:34.306976+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:37.075 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-16T11:49:37.075 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:17.360859+0000', 'last_change': '2023-12-16T11:44:34.316813+0000', 'last_active': '2023-12-16T11:47:17.360859+0000', 'last_peered': '2023-12-16T11:47:17.360859+0000', 'last_clean': '2023-12-16T11:44:21.274733+0000', 'last_became_active': '2023-12-16T11:44:34.316813+0000', 'last_became_peered': '2023-12-16T11:44:34.316813+0000', 'last_unstale': '2023-12-16T11:47:17.360859+0000', 'last_undegraded': '2023-12-16T11:47:17.360859+0000', 'last_fullsized': '2023-12-16T11:44:34.302945+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:37.076 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-16T11:49:37.076 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 400, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:47:12.360326+0000', 'last_change': '2023-12-16T11:44:34.315229+0000', 'last_active': '2023-12-16T11:47:12.360326+0000', 'last_peered': '2023-12-16T11:47:12.360326+0000', 'last_clean': '2023-12-16T11:44:21.274477+0000', 'last_became_active': '2023-12-16T11:44:34.315229+0000', 'last_became_peered': '2023-12-16T11:44:34.315229+0000', 'last_unstale': '2023-12-16T11:47:12.360326+0000', 'last_undegraded': '2023-12-16T11:47:12.360326+0000', 'last_fullsized': '2023-12-16T11:44:34.302957+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:37.076 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-16T11:49:37.076 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 380, 'state': 'active+undersized', 'last_fresh': '2023-12-16T11:46:42.357070+0000', 'last_change': '2023-12-16T11:44:34.321612+0000', 'last_active': '2023-12-16T11:46:42.357070+0000', 'last_peered': '2023-12-16T11:46:42.357070+0000', 'last_clean': '2023-12-16T11:09:38.367601+0000', 'last_became_active': '2023-12-16T11:44:34.321612+0000', 'last_became_peered': '2023-12-16T11:44:34.321612+0000', 'last_unstale': '2023-12-16T11:46:42.357070+0000', 'last_undegraded': '2023-12-16T11:46:42.357070+0000', 'last_fullsized': '2023-12-16T11:44:34.306677+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:38.367601+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:37.076 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-16T11:49:37.076 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "10'97", 'reported_seq': 530, 'reported_epoch': 380, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-16T11:45:42.350396+0000', 'last_change': '2023-12-16T11:44:34.308356+0000', 'last_active': '2023-12-16T11:45:42.350396+0000', 'last_peered': '2023-12-16T11:45:42.350396+0000', 'last_clean': '2023-12-16T11:44:21.273097+0000', 'last_became_active': '2023-12-16T11:44:34.308356+0000', 'last_became_peered': '2023-12-16T11:44:34.308356+0000', 'last_unstale': '2023-12-16T11:45:42.350396+0000', 'last_undegraded': '2023-12-16T11:44:34.299204+0000', 'last_fullsized': '2023-12-16T11:44:34.299091+0000', 'mapping_epoch': 376, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'last_clean_scrub_stamp': '2023-12-16T11:09:32.494184+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-16T11:49:37.076 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-16T11:49:37.417 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:37.417 INFO:teuthology.orchestra.run.smithi160.stdout:{"fsid":"fdb8828c-a0b4-49f4-bc4d-e8356adcab23","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2415,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":380,"num_osds":3,"num_up_osds":2,"osd_up_since":1702727073,"num_in_osds":3,"osd_in_since":1702724963,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3737542656,"bytes_avail":286172749824,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":38,"modified":"2023-12-16T11:45:18.236175+0000","services":{}},"progress_events":{}} 2023-12-16T11:49:37.430 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-16T11:49:37.430 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:49:37.907 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-12-16T11:49:37.907 INFO:teuthology.orchestra.run.smithi160.stderr:dumped all 2023-12-16T11:49:37.919 INFO:teuthology.orchestra.run.smithi160.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-12-16T11:49:36.293898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3649944,"kb_used_data":419292,"kb_used_omap":293,"kb_used_meta":3230234,"kb_avail":279465576,"statfs":{"total":289910292480,"available":286172749824,"internally_reserved":0,"allocated":429355008,"data_stored":424781927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":300614,"internal_metadata":3307760058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002836"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:32.355808+0000","last_change":"2023-12-16T11:44:34.332009+0000","last_active":"2023-12-16T11:46:32.355808+0000","last_peered":"2023-12-16T11:46:32.355808+0000","last_clean":"2023-12-16T11:44:22.322939+0000","last_became_active":"2023-12-16T11:44:34.332009+0000","last_became_peered":"2023-12-16T11:44:34.332009+0000","last_unstale":"2023-12-16T11:46:32.355808+0000","last_undegraded":"2023-12-16T11:46:32.355808+0000","last_fullsized":"2023-12-16T11:44:34.319771+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:49:29.692430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:27.355268+0000","last_change":"2023-12-16T11:44:34.317288+0000","last_active":"2023-12-16T11:46:27.355268+0000","last_peered":"2023-12-16T11:46:27.355268+0000","last_clean":"2023-12-16T11:44:21.274929+0000","last_became_active":"2023-12-16T11:44:34.317288+0000","last_became_peered":"2023-12-16T11:44:34.317288+0000","last_unstale":"2023-12-16T11:46:27.355268+0000","last_undegraded":"2023-12-16T11:46:27.355268+0000","last_fullsized":"2023-12-16T11:44:34.306976+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:42:14.815290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:17.360859+0000","last_change":"2023-12-16T11:44:34.316813+0000","last_active":"2023-12-16T11:47:17.360859+0000","last_peered":"2023-12-16T11:47:17.360859+0000","last_clean":"2023-12-16T11:44:21.274733+0000","last_became_active":"2023-12-16T11:44:34.316813+0000","last_became_peered":"2023-12-16T11:44:34.316813+0000","last_unstale":"2023-12-16T11:47:17.360859+0000","last_undegraded":"2023-12-16T11:47:17.360859+0000","last_fullsized":"2023-12-16T11:44:34.302945+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:19:57.639385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:47:12.360326+0000","last_change":"2023-12-16T11:44:34.315229+0000","last_active":"2023-12-16T11:47:12.360326+0000","last_peered":"2023-12-16T11:47:12.360326+0000","last_clean":"2023-12-16T11:44:21.274477+0000","last_became_active":"2023-12-16T11:44:34.315229+0000","last_became_peered":"2023-12-16T11:44:34.315229+0000","last_unstale":"2023-12-16T11:47:12.360326+0000","last_undegraded":"2023-12-16T11:47:12.360326+0000","last_fullsized":"2023-12-16T11:44:34.302957+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:17:54.837437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":380,"state":"active+undersized","last_fresh":"2023-12-16T11:46:42.357070+0000","last_change":"2023-12-16T11:44:34.321612+0000","last_active":"2023-12-16T11:46:42.357070+0000","last_peered":"2023-12-16T11:46:42.357070+0000","last_clean":"2023-12-16T11:09:38.367601+0000","last_became_active":"2023-12-16T11:44:34.321612+0000","last_became_peered":"2023-12-16T11:44:34.321612+0000","last_unstale":"2023-12-16T11:46:42.357070+0000","last_undegraded":"2023-12-16T11:46:42.357070+0000","last_fullsized":"2023-12-16T11:44:34.306677+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:43:46.893562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:07.108176+0000","last_change":"2023-12-16T11:09:39.382260+0000","last_active":"2023-12-16T11:47:07.108176+0000","last_peered":"2023-12-16T11:47:07.108176+0000","last_clean":"2023-12-16T11:47:07.108176+0000","last_became_active":"2023-12-16T11:09:39.381956+0000","last_became_peered":"2023-12-16T11:09:39.381956+0000","last_unstale":"2023-12-16T11:47:07.108176+0000","last_undegraded":"2023-12-16T11:47:07.108176+0000","last_fullsized":"2023-12-16T11:47:07.108176+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T14:13:04.095136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:45:47.100049+0000","last_change":"2023-12-16T11:09:39.383868+0000","last_active":"2023-12-16T11:45:47.100049+0000","last_peered":"2023-12-16T11:45:47.100049+0000","last_clean":"2023-12-16T11:45:47.100049+0000","last_became_active":"2023-12-16T11:09:39.383417+0000","last_became_peered":"2023-12-16T11:09:39.383417+0000","last_unstale":"2023-12-16T11:45:47.100049+0000","last_undegraded":"2023-12-16T11:45:47.100049+0000","last_fullsized":"2023-12-16T11:45:47.100049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T17:51:15.384625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":396,"reported_epoch":380,"state":"active+clean","last_fresh":"2023-12-16T11:47:12.360300+0000","last_change":"2023-12-16T11:09:39.381625+0000","last_active":"2023-12-16T11:47:12.360300+0000","last_peered":"2023-12-16T11:47:12.360300+0000","last_clean":"2023-12-16T11:47:12.360300+0000","last_became_active":"2023-12-16T11:09:39.381354+0000","last_became_peered":"2023-12-16T11:09:39.381354+0000","last_unstale":"2023-12-16T11:47:12.360300+0000","last_undegraded":"2023-12-16T11:47:12.360300+0000","last_fullsized":"2023-12-16T11:47:12.360300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:38.367601+0000","last_clean_scrub_stamp":"2023-12-16T11:09:38.367601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:48:59.461543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":530,"reported_epoch":380,"state":"active+undersized+degraded","last_fresh":"2023-12-16T11:45:42.350396+0000","last_change":"2023-12-16T11:44:34.308356+0000","last_active":"2023-12-16T11:45:42.350396+0000","last_peered":"2023-12-16T11:45:42.350396+0000","last_clean":"2023-12-16T11:44:21.273097+0000","last_became_active":"2023-12-16T11:44:34.308356+0000","last_became_peered":"2023-12-16T11:44:34.308356+0000","last_unstale":"2023-12-16T11:45:42.350396+0000","last_undegraded":"2023-12-16T11:44:34.299204+0000","last_fullsized":"2023-12-16T11:44:34.299091+0000","mapping_epoch":376,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:32.494184+0000","last_clean_scrub_stamp":"2023-12-16T11:09:32.494184+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:00:40.039867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":830844,"kb_used_data":3056,"kb_used_omap":90,"kb_used_meta":827685,"kb_avail":93540996,"statfs":{"total":96636764160,"available":95785979904,"internally_reserved":0,"allocated":3129344,"data_stored":1676559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92580,"internal_metadata":847550044},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.41299999999999998},"min":{"1min":0.22700000000000001,"5min":0.159,"15min":0.14499999999999999},"max":{"1min":3.827,"5min":5.2990000000000004,"15min":5.2990000000000004},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53200000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.20799999999999999,"15min":0.155},"max":{"1min":3.8620000000000001,"5min":5.3319999999999999,"15min":5.3319999999999999},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Dec 16 11:48:50 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.27000000000000002,"15min":0.379},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.45800000000000002,"5min":0.47199999999999998,"15min":5.2119999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.29899999999999999,"15min":0.41499999999999998},"min":{"1min":0.23499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.436,"5min":0.95999999999999996,"15min":5.391},"last":0.26100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017792,"kb_used_data":3636,"kb_used_omap":94,"kb_used_meta":1014049,"kb_avail":93354048,"statfs":{"total":96636764160,"available":95594545152,"internally_reserved":0,"allocated":3723264,"data_stored":2266927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97074,"internal_metadata":1038386382},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.504,"15min":0.40899999999999997},"min":{"1min":0.187,"5min":0.151,"15min":0.14299999999999999},"max":{"1min":2.9340000000000002,"5min":7.8230000000000004,"15min":7.8230000000000004},"last":0.188},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.45900000000000002},"min":{"1min":0.21199999999999999,"5min":0.188,"15min":0.17699999999999999},"max":{"1min":2.125,"5min":5.173,"15min":14.074},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 11:48:43 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.44700000000000001,"15min":0.71999999999999997},"min":{"1min":0.182,"5min":0.158,"15min":0.158},"max":{"1min":0.36299999999999999,"5min":9.4480000000000004,"15min":74.266000000000005},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.31,"5min":0.52400000000000002,"15min":0.68200000000000005},"min":{"1min":0.216,"5min":0.214,"15min":0.20200000000000001},"max":{"1min":0.42799999999999999,"5min":11.497,"15min":73.727000000000004},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1801308,"kb_used_data":412600,"kb_used_omap":108,"kb_used_meta":1388499,"kb_avail":92570532,"statfs":{"total":96636764160,"available":94792224768,"internally_reserved":0,"allocated":422502400,"data_stored":420838441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110960,"internal_metadata":1421823632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.45600000000000002,"15min":0.52600000000000002},"min":{"1min":0.183,"5min":0.153,"15min":0.13500000000000001},"max":{"1min":1.022,"5min":8.25,"15min":47.898000000000003},"last":2.407},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54500000000000004,"15min":0.56999999999999995},"min":{"1min":0.216,"5min":0.182,"15min":0.159},"max":{"1min":3.7639999999999998,"5min":8.2919999999999998,"15min":47.872},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.78000000000000003,"15min":0.63900000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.159},"max":{"1min":4.1470000000000002,"5min":14.118,"15min":47.853999999999999},"last":1.216},{"interface":"front","average":{"1min":1.5149999999999999,"5min":0.72499999999999998,"15min":0.627},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.782,"5min":18.782,"15min":47.222000000000001},"last":0.71699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:49:39.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-12-16T11:49:39.540 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-16T11:49:39.944 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-12-16T11:49:39.944 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-16T11:49:39.944 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-16T11:49:39.945 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-16T11:49:39.945 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:49:39.943+0000 7ff3f696f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 85923) UID: 0 2023-12-16T11:49:39.945 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:49:39.943+0000 7ff3f696f640 -1 osd.1 380 *** Got signal Terminated *** 2023-12-16T11:49:39.945 INFO:tasks.ceph.cluster2.osd.1.smithi160.stderr:2023-12-16T11:49:39.943+0000 7ff3f696f640 -1 osd.1 380 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T11:49:39.946 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:49:39.943+0000 7f14050e3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 85925) UID: 0 2023-12-16T11:49:39.946 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:49:39.943+0000 7f14050e3640 -1 osd.2 380 *** Got signal Terminated *** 2023-12-16T11:49:39.946 INFO:tasks.ceph.cluster2.osd.2.smithi160.stderr:2023-12-16T11:49:39.943+0000 7f14050e3640 -1 osd.2 380 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T11:49:39.946 INFO:tasks.ceph.cluster2.mon.a.smithi160.stderr:2023-12-16T11:49:39.943+0000 7fd86769e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 85615) UID: 0 2023-12-16T11:49:39.946 INFO:tasks.ceph.cluster2.mon.a.smithi160.stderr:2023-12-16T11:49:39.943+0000 7fd86769e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-16T11:49:39.946 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:49:39.944+0000 7fa35f650640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 90513) UID: 0 2023-12-16T11:49:39.947 INFO:tasks.ceph.cluster2.mgr.x.smithi087.stderr:2023-12-16T11:49:39.944+0000 7fa35f650640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-16T11:49:40.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-16T11:51:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T11:51:40.964 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-16T11:51:40.966 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-16T11:51:40.967 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-16T11:51:40.967 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-12-16T11:51:40.967 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:51:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-16T11:51:40.967 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-12-16T11:51:40.968 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-12-16T11:51:40.968 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-12-16T11:51:40.968 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:51:40.968 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-12-16T11:51:40.968 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-12-16T11:51:40.968 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:51:40.968 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-12-16T11:51:40.968 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-16T11:51:40.969 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-12-16T11:51:40.969 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:51:40.969 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-12-16T11:51:40.969 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-16T11:51:40.969 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-12-16T11:51:40.969 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:51:40.969 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-12-16T11:51:40.969 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-16T11:51:40.970 DEBUG:teuthology.orchestra.run.smithi160:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-16T11:51:41.004 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:51:41.004 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-12-16T11:51:41.272 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:51:41.273 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-12-16T11:51:41.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:51:41.436 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-12-16T11:51:41.588 INFO:tasks.ceph:Archiving mon data... 2023-12-16T11:51:41.589 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/data/cluster2.mon.a.tgz 2023-12-16T11:51:41.590 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2023-12-16T11:51:41.604 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.K2GE92Kexh 2023-12-16T11:51:41.605 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.K2GE92Kexh 2023-12-16T11:51:41.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.K2GE92Kexh 2023-12-16T11:51:41.830 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.K2GE92Kexh is 322KB 2023-12-16T11:51:41.927 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.K2GE92Kexh 2023-12-16T11:51:41.941 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-16T11:51:41.942 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-16T11:51:41.944 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-16T11:51:42.007 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-16T11:51:42.008 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-16T11:51:42.020 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-16T11:51:42.021 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-16T11:51:42.306 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:51:42.306 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-16T11:51:42.318 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1740,"stamp":"2023-12-16T11:51:40.871745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312416,"kb_used_data":12220,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282803104,"statfs":{"total":289910292480,"available":289590378496,"internally_reserved":0,"allocated":12513280,"data_stored":7270105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":307359055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002418"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:06.077280+0000","last_change":"2023-12-16T11:09:03.005273+0000","last_active":"2023-12-16T11:48:06.077280+0000","last_peered":"2023-12-16T11:48:06.077280+0000","last_clean":"2023-12-16T11:48:06.077280+0000","last_became_active":"2023-12-16T11:09:03.004893+0000","last_became_peered":"2023-12-16T11:09:03.004893+0000","last_unstale":"2023-12-16T11:48:06.077280+0000","last_undegraded":"2023-12-16T11:48:06.077280+0000","last_fullsized":"2023-12-16T11:48:06.077280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:41:43.307235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:01.076765+0000","last_change":"2023-12-16T11:09:03.005695+0000","last_active":"2023-12-16T11:48:01.076765+0000","last_peered":"2023-12-16T11:48:01.076765+0000","last_clean":"2023-12-16T11:48:01.076765+0000","last_became_active":"2023-12-16T11:09:03.005445+0000","last_became_peered":"2023-12-16T11:09:03.005445+0000","last_unstale":"2023-12-16T11:48:01.076765+0000","last_undegraded":"2023-12-16T11:48:01.076765+0000","last_fullsized":"2023-12-16T11:48:01.076765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:17:51.300986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:51.082529+0000","last_change":"2023-12-16T11:09:03.005110+0000","last_active":"2023-12-16T11:48:51.082529+0000","last_peered":"2023-12-16T11:48:51.082529+0000","last_clean":"2023-12-16T11:48:51.082529+0000","last_became_active":"2023-12-16T11:09:03.004818+0000","last_became_peered":"2023-12-16T11:09:03.004818+0000","last_unstale":"2023-12-16T11:48:51.082529+0000","last_undegraded":"2023-12-16T11:48:51.082529+0000","last_fullsized":"2023-12-16T11:48:51.082529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:13:30.010815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081386+0000","last_change":"2023-12-16T11:09:03.004551+0000","last_active":"2023-12-16T11:48:41.081386+0000","last_peered":"2023-12-16T11:48:41.081386+0000","last_clean":"2023-12-16T11:48:41.081386+0000","last_became_active":"2023-12-16T11:09:03.004235+0000","last_became_peered":"2023-12-16T11:09:03.004235+0000","last_unstale":"2023-12-16T11:48:41.081386+0000","last_undegraded":"2023-12-16T11:48:41.081386+0000","last_fullsized":"2023-12-16T11:48:41.081386+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:25:33.320035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":474,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.804399+0000","last_change":"2023-12-16T11:09:03.006609+0000","last_active":"2023-12-16T11:48:41.804399+0000","last_peered":"2023-12-16T11:48:41.804399+0000","last_clean":"2023-12-16T11:48:41.804399+0000","last_became_active":"2023-12-16T11:09:03.006307+0000","last_became_peered":"2023-12-16T11:09:03.006307+0000","last_unstale":"2023-12-16T11:48:41.804399+0000","last_undegraded":"2023-12-16T11:48:41.804399+0000","last_fullsized":"2023-12-16T11:48:41.804399+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:33:31.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:36.265665+0000","last_change":"2023-12-16T11:09:03.004443+0000","last_active":"2023-12-16T11:48:36.265665+0000","last_peered":"2023-12-16T11:48:36.265665+0000","last_clean":"2023-12-16T11:48:36.265665+0000","last_became_active":"2023-12-16T11:09:03.004159+0000","last_became_peered":"2023-12-16T11:09:03.004159+0000","last_unstale":"2023-12-16T11:48:36.265665+0000","last_undegraded":"2023-12-16T11:48:36.265665+0000","last_fullsized":"2023-12-16T11:48:36.265665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:10:30.195137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.269544+0000","last_change":"2023-12-16T11:09:03.004009+0000","last_active":"2023-12-16T11:49:11.269544+0000","last_peered":"2023-12-16T11:49:11.269544+0000","last_clean":"2023-12-16T11:49:11.269544+0000","last_became_active":"2023-12-16T11:09:03.003715+0000","last_became_peered":"2023-12-16T11:09:03.003715+0000","last_unstale":"2023-12-16T11:49:11.269544+0000","last_undegraded":"2023-12-16T11:49:11.269544+0000","last_fullsized":"2023-12-16T11:49:11.269544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:12:11.654608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081360+0000","last_change":"2023-12-16T11:09:03.006746+0000","last_active":"2023-12-16T11:48:41.081360+0000","last_peered":"2023-12-16T11:48:41.081360+0000","last_clean":"2023-12-16T11:48:41.081360+0000","last_became_active":"2023-12-16T11:09:03.006304+0000","last_became_peered":"2023-12-16T11:09:03.006304+0000","last_unstale":"2023-12-16T11:48:41.081360+0000","last_undegraded":"2023-12-16T11:48:41.081360+0000","last_fullsized":"2023-12-16T11:48:41.081360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:20:34.752327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.084597+0000","last_change":"2023-12-16T11:08:58.062810+0000","last_active":"2023-12-16T11:49:11.084597+0000","last_peered":"2023-12-16T11:49:11.084597+0000","last_clean":"2023-12-16T11:49:11.084597+0000","last_became_active":"2023-12-16T11:08:58.061796+0000","last_became_peered":"2023-12-16T11:08:58.061796+0000","last_unstale":"2023-12-16T11:49:11.084597+0000","last_undegraded":"2023-12-16T11:49:11.084597+0000","last_fullsized":"2023-12-16T11:49:11.084597+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_clean_scrub_stamp":"2023-12-16T11:08:56.977736+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:33:16.597547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97268,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274572,"statfs":{"total":96636764160,"available":96537161728,"internally_reserved":0,"allocated":3776512,"data_stored":2030046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.375,"5min":0.50800000000000001,"15min":0.95699999999999996},"last":0.255},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.19700000000000001},"max":{"1min":0.42699999999999999,"5min":0.54200000000000004,"15min":0.80800000000000005},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.17899999999999999},"max":{"1min":0.42799999999999999,"5min":0.52500000000000002,"15min":0.626},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.35499999999999998,"15min":0.33600000000000002},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.55700000000000005,"15min":0.90900000000000003},"last":0.373}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93688,"kb_used_data":4268,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278152,"statfs":{"total":96636764160,"available":96540827648,"internally_reserved":0,"allocated":4370432,"data_stored":2620414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32400000000000001,"15min":0.313},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.184},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.31},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.45300000000000001,"5min":0.51000000000000001,"15min":0.58699999999999997},"last":0.33300000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":0.504,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.496,"15min":0.51500000000000001},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121460,"kb_used_data":4264,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250380,"statfs":{"total":96636764160,"available":96512389120,"internally_reserved":0,"allocated":4366336,"data_stored":2619645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.312,"15min":0.30599999999999999},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.16900000000000001},"max":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38300000000000001,"15min":0.34699999999999998},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.183},"max":{"1min":0.48999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.222},"max":{"1min":0.45300000000000001,"5min":0.49099999999999999,"15min":0.51800000000000002},"last":0.378},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42299999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:51:42.319 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-16T11:51:42.594 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:51:42.594 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-16T11:51:42.606 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1740,"stamp":"2023-12-16T11:51:40.871745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312416,"kb_used_data":12220,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282803104,"statfs":{"total":289910292480,"available":289590378496,"internally_reserved":0,"allocated":12513280,"data_stored":7270105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":307359055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002418"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:06.077280+0000","last_change":"2023-12-16T11:09:03.005273+0000","last_active":"2023-12-16T11:48:06.077280+0000","last_peered":"2023-12-16T11:48:06.077280+0000","last_clean":"2023-12-16T11:48:06.077280+0000","last_became_active":"2023-12-16T11:09:03.004893+0000","last_became_peered":"2023-12-16T11:09:03.004893+0000","last_unstale":"2023-12-16T11:48:06.077280+0000","last_undegraded":"2023-12-16T11:48:06.077280+0000","last_fullsized":"2023-12-16T11:48:06.077280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:41:43.307235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:01.076765+0000","last_change":"2023-12-16T11:09:03.005695+0000","last_active":"2023-12-16T11:48:01.076765+0000","last_peered":"2023-12-16T11:48:01.076765+0000","last_clean":"2023-12-16T11:48:01.076765+0000","last_became_active":"2023-12-16T11:09:03.005445+0000","last_became_peered":"2023-12-16T11:09:03.005445+0000","last_unstale":"2023-12-16T11:48:01.076765+0000","last_undegraded":"2023-12-16T11:48:01.076765+0000","last_fullsized":"2023-12-16T11:48:01.076765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:17:51.300986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:51.082529+0000","last_change":"2023-12-16T11:09:03.005110+0000","last_active":"2023-12-16T11:48:51.082529+0000","last_peered":"2023-12-16T11:48:51.082529+0000","last_clean":"2023-12-16T11:48:51.082529+0000","last_became_active":"2023-12-16T11:09:03.004818+0000","last_became_peered":"2023-12-16T11:09:03.004818+0000","last_unstale":"2023-12-16T11:48:51.082529+0000","last_undegraded":"2023-12-16T11:48:51.082529+0000","last_fullsized":"2023-12-16T11:48:51.082529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:13:30.010815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081386+0000","last_change":"2023-12-16T11:09:03.004551+0000","last_active":"2023-12-16T11:48:41.081386+0000","last_peered":"2023-12-16T11:48:41.081386+0000","last_clean":"2023-12-16T11:48:41.081386+0000","last_became_active":"2023-12-16T11:09:03.004235+0000","last_became_peered":"2023-12-16T11:09:03.004235+0000","last_unstale":"2023-12-16T11:48:41.081386+0000","last_undegraded":"2023-12-16T11:48:41.081386+0000","last_fullsized":"2023-12-16T11:48:41.081386+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:25:33.320035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":474,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.804399+0000","last_change":"2023-12-16T11:09:03.006609+0000","last_active":"2023-12-16T11:48:41.804399+0000","last_peered":"2023-12-16T11:48:41.804399+0000","last_clean":"2023-12-16T11:48:41.804399+0000","last_became_active":"2023-12-16T11:09:03.006307+0000","last_became_peered":"2023-12-16T11:09:03.006307+0000","last_unstale":"2023-12-16T11:48:41.804399+0000","last_undegraded":"2023-12-16T11:48:41.804399+0000","last_fullsized":"2023-12-16T11:48:41.804399+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:33:31.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:36.265665+0000","last_change":"2023-12-16T11:09:03.004443+0000","last_active":"2023-12-16T11:48:36.265665+0000","last_peered":"2023-12-16T11:48:36.265665+0000","last_clean":"2023-12-16T11:48:36.265665+0000","last_became_active":"2023-12-16T11:09:03.004159+0000","last_became_peered":"2023-12-16T11:09:03.004159+0000","last_unstale":"2023-12-16T11:48:36.265665+0000","last_undegraded":"2023-12-16T11:48:36.265665+0000","last_fullsized":"2023-12-16T11:48:36.265665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:10:30.195137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.269544+0000","last_change":"2023-12-16T11:09:03.004009+0000","last_active":"2023-12-16T11:49:11.269544+0000","last_peered":"2023-12-16T11:49:11.269544+0000","last_clean":"2023-12-16T11:49:11.269544+0000","last_became_active":"2023-12-16T11:09:03.003715+0000","last_became_peered":"2023-12-16T11:09:03.003715+0000","last_unstale":"2023-12-16T11:49:11.269544+0000","last_undegraded":"2023-12-16T11:49:11.269544+0000","last_fullsized":"2023-12-16T11:49:11.269544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:12:11.654608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081360+0000","last_change":"2023-12-16T11:09:03.006746+0000","last_active":"2023-12-16T11:48:41.081360+0000","last_peered":"2023-12-16T11:48:41.081360+0000","last_clean":"2023-12-16T11:48:41.081360+0000","last_became_active":"2023-12-16T11:09:03.006304+0000","last_became_peered":"2023-12-16T11:09:03.006304+0000","last_unstale":"2023-12-16T11:48:41.081360+0000","last_undegraded":"2023-12-16T11:48:41.081360+0000","last_fullsized":"2023-12-16T11:48:41.081360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:20:34.752327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.084597+0000","last_change":"2023-12-16T11:08:58.062810+0000","last_active":"2023-12-16T11:49:11.084597+0000","last_peered":"2023-12-16T11:49:11.084597+0000","last_clean":"2023-12-16T11:49:11.084597+0000","last_became_active":"2023-12-16T11:08:58.061796+0000","last_became_peered":"2023-12-16T11:08:58.061796+0000","last_unstale":"2023-12-16T11:49:11.084597+0000","last_undegraded":"2023-12-16T11:49:11.084597+0000","last_fullsized":"2023-12-16T11:49:11.084597+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_clean_scrub_stamp":"2023-12-16T11:08:56.977736+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:33:16.597547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738886,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97268,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274572,"statfs":{"total":96636764160,"available":96537161728,"internally_reserved":0,"allocated":3776512,"data_stored":2030046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.375,"5min":0.50800000000000001,"15min":0.95699999999999996},"last":0.255},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.19700000000000001},"max":{"1min":0.42699999999999999,"5min":0.54200000000000004,"15min":0.80800000000000005},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.17899999999999999},"max":{"1min":0.42799999999999999,"5min":0.52500000000000002,"15min":0.626},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.35499999999999998,"15min":0.33600000000000002},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.55700000000000005,"15min":0.90900000000000003},"last":0.373}]}]},{"osd":1,"up_from":8,"seq":34359738884,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93688,"kb_used_data":4268,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278152,"statfs":{"total":96636764160,"available":96540827648,"internally_reserved":0,"allocated":4370432,"data_stored":2620414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32400000000000001,"15min":0.313},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.184},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.31},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.45300000000000001,"5min":0.51000000000000001,"15min":0.58699999999999997},"last":0.33300000000000002}]},{"osd":2,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":0.504,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.496,"15min":0.51500000000000001},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738885,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121460,"kb_used_data":4264,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250380,"statfs":{"total":96636764160,"available":96512389120,"internally_reserved":0,"allocated":4366336,"data_stored":2619645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.312,"15min":0.30599999999999999},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.16900000000000001},"max":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38300000000000001,"15min":0.34699999999999998},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.183},"max":{"1min":0.48999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.222},"max":{"1min":0.45300000000000001,"5min":0.49099999999999999,"15min":0.51800000000000002},"last":0.378},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42299999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:51:42.607 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-16T11:51:42.607 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-16T11:51:42.880 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:51:42.880 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-16T11:51:42.892 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1741,"stamp":"2023-12-16T11:51:42.872153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312416,"kb_used_data":12220,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282803104,"statfs":{"total":289910292480,"available":289590378496,"internally_reserved":0,"allocated":12513280,"data_stored":7270105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":307359055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002374"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:06.077280+0000","last_change":"2023-12-16T11:09:03.005273+0000","last_active":"2023-12-16T11:48:06.077280+0000","last_peered":"2023-12-16T11:48:06.077280+0000","last_clean":"2023-12-16T11:48:06.077280+0000","last_became_active":"2023-12-16T11:09:03.004893+0000","last_became_peered":"2023-12-16T11:09:03.004893+0000","last_unstale":"2023-12-16T11:48:06.077280+0000","last_undegraded":"2023-12-16T11:48:06.077280+0000","last_fullsized":"2023-12-16T11:48:06.077280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:41:43.307235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:01.076765+0000","last_change":"2023-12-16T11:09:03.005695+0000","last_active":"2023-12-16T11:48:01.076765+0000","last_peered":"2023-12-16T11:48:01.076765+0000","last_clean":"2023-12-16T11:48:01.076765+0000","last_became_active":"2023-12-16T11:09:03.005445+0000","last_became_peered":"2023-12-16T11:09:03.005445+0000","last_unstale":"2023-12-16T11:48:01.076765+0000","last_undegraded":"2023-12-16T11:48:01.076765+0000","last_fullsized":"2023-12-16T11:48:01.076765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:17:51.300986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:51.082529+0000","last_change":"2023-12-16T11:09:03.005110+0000","last_active":"2023-12-16T11:48:51.082529+0000","last_peered":"2023-12-16T11:48:51.082529+0000","last_clean":"2023-12-16T11:48:51.082529+0000","last_became_active":"2023-12-16T11:09:03.004818+0000","last_became_peered":"2023-12-16T11:09:03.004818+0000","last_unstale":"2023-12-16T11:48:51.082529+0000","last_undegraded":"2023-12-16T11:48:51.082529+0000","last_fullsized":"2023-12-16T11:48:51.082529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:13:30.010815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081386+0000","last_change":"2023-12-16T11:09:03.004551+0000","last_active":"2023-12-16T11:48:41.081386+0000","last_peered":"2023-12-16T11:48:41.081386+0000","last_clean":"2023-12-16T11:48:41.081386+0000","last_became_active":"2023-12-16T11:09:03.004235+0000","last_became_peered":"2023-12-16T11:09:03.004235+0000","last_unstale":"2023-12-16T11:48:41.081386+0000","last_undegraded":"2023-12-16T11:48:41.081386+0000","last_fullsized":"2023-12-16T11:48:41.081386+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:25:33.320035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":474,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.804399+0000","last_change":"2023-12-16T11:09:03.006609+0000","last_active":"2023-12-16T11:48:41.804399+0000","last_peered":"2023-12-16T11:48:41.804399+0000","last_clean":"2023-12-16T11:48:41.804399+0000","last_became_active":"2023-12-16T11:09:03.006307+0000","last_became_peered":"2023-12-16T11:09:03.006307+0000","last_unstale":"2023-12-16T11:48:41.804399+0000","last_undegraded":"2023-12-16T11:48:41.804399+0000","last_fullsized":"2023-12-16T11:48:41.804399+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:33:31.269875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:36.265665+0000","last_change":"2023-12-16T11:09:03.004443+0000","last_active":"2023-12-16T11:48:36.265665+0000","last_peered":"2023-12-16T11:48:36.265665+0000","last_clean":"2023-12-16T11:48:36.265665+0000","last_became_active":"2023-12-16T11:09:03.004159+0000","last_became_peered":"2023-12-16T11:09:03.004159+0000","last_unstale":"2023-12-16T11:48:36.265665+0000","last_undegraded":"2023-12-16T11:48:36.265665+0000","last_fullsized":"2023-12-16T11:48:36.265665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:10:30.195137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.269544+0000","last_change":"2023-12-16T11:09:03.004009+0000","last_active":"2023-12-16T11:49:11.269544+0000","last_peered":"2023-12-16T11:49:11.269544+0000","last_clean":"2023-12-16T11:49:11.269544+0000","last_became_active":"2023-12-16T11:09:03.003715+0000","last_became_peered":"2023-12-16T11:09:03.003715+0000","last_unstale":"2023-12-16T11:49:11.269544+0000","last_undegraded":"2023-12-16T11:49:11.269544+0000","last_fullsized":"2023-12-16T11:49:11.269544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:12:11.654608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081360+0000","last_change":"2023-12-16T11:09:03.006746+0000","last_active":"2023-12-16T11:48:41.081360+0000","last_peered":"2023-12-16T11:48:41.081360+0000","last_clean":"2023-12-16T11:48:41.081360+0000","last_became_active":"2023-12-16T11:09:03.006304+0000","last_became_peered":"2023-12-16T11:09:03.006304+0000","last_unstale":"2023-12-16T11:48:41.081360+0000","last_undegraded":"2023-12-16T11:48:41.081360+0000","last_fullsized":"2023-12-16T11:48:41.081360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:20:34.752327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.084597+0000","last_change":"2023-12-16T11:08:58.062810+0000","last_active":"2023-12-16T11:49:11.084597+0000","last_peered":"2023-12-16T11:49:11.084597+0000","last_clean":"2023-12-16T11:49:11.084597+0000","last_became_active":"2023-12-16T11:08:58.061796+0000","last_became_peered":"2023-12-16T11:08:58.061796+0000","last_unstale":"2023-12-16T11:49:11.084597+0000","last_undegraded":"2023-12-16T11:49:11.084597+0000","last_fullsized":"2023-12-16T11:49:11.084597+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_clean_scrub_stamp":"2023-12-16T11:08:56.977736+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:33:16.597547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97268,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274572,"statfs":{"total":96636764160,"available":96537161728,"internally_reserved":0,"allocated":3776512,"data_stored":2030046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.375,"5min":0.50800000000000001,"15min":0.95699999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.19700000000000001},"max":{"1min":0.42699999999999999,"5min":0.54200000000000004,"15min":0.80800000000000005},"last":0.39800000000000002}]},{"osd":1,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.17899999999999999},"max":{"1min":0.42799999999999999,"5min":0.52500000000000002,"15min":0.626},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.35499999999999998,"15min":0.33600000000000002},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.55700000000000005,"15min":0.90900000000000003},"last":0.45800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93688,"kb_used_data":4268,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278152,"statfs":{"total":96636764160,"available":96540827648,"internally_reserved":0,"allocated":4370432,"data_stored":2620414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32400000000000001,"15min":0.313},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.184},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.45300000000000001,"5min":0.51000000000000001,"15min":0.58699999999999997},"last":0.23400000000000001}]},{"osd":2,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":0.504,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.496,"15min":0.51500000000000001},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121460,"kb_used_data":4264,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250380,"statfs":{"total":96636764160,"available":96512389120,"internally_reserved":0,"allocated":4366336,"data_stored":2619645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.312,"15min":0.30599999999999999},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.16900000000000001},"max":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.251},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38300000000000001,"15min":0.34699999999999998},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.183},"max":{"1min":0.48999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.34799999999999998}]},{"osd":2,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.222},"max":{"1min":0.45300000000000001,"5min":0.49099999999999999,"15min":0.51800000000000002},"last":0.312},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42299999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:51:42.893 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-16T11:51:43.170 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:51:43.170 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":459,"fsid":"f15fcecc-2053-424b-9728-3322fd59761d","created":"2023-12-16T11:08:46.007497+0000","modified":"2023-12-16T11:45:15.396624+0000","last_up_change":"2023-12-16T11:08:55.788211+0000","last_in_change":"2023-12-16T11:08:46.857634+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T11:08:56.289896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-16T11:09:01.171025+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"48068439-5b2f-425e-80db-df998bcb0fca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6807","nonce":2153554517}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6813","nonce":2153554517}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6821","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6823","nonce":2153554517}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":2153554517},{"type":"v1","addr":"172.21.15.87:6819","nonce":2153554517}]},"public_addr":"172.21.15.87:6807/2153554517","cluster_addr":"172.21.15.87:6813/2153554517","heartbeat_back_addr":"172.21.15.87:6823/2153554517","heartbeat_front_addr":"172.21.15.87:6819/2153554517","state":["exists","up"]},{"osd":1,"uuid":"df567ad0-a71a-478c-9675-70c482bc93d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6801","nonce":3512712324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6806","nonce":3512712324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6818","nonce":3512712324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3512712324},{"type":"v1","addr":"172.21.15.87:6811","nonce":3512712324}]},"public_addr":"172.21.15.87:6801/3512712324","cluster_addr":"172.21.15.87:6806/3512712324","heartbeat_back_addr":"172.21.15.87:6818/3512712324","heartbeat_front_addr":"172.21.15.87:6811/3512712324","state":["exists","up"]},{"osd":2,"uuid":"5a05331c-9130-4466-92a6-6661928b0926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6805","nonce":1086837511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6812","nonce":1086837511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6822","nonce":1086837511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":1086837511},{"type":"v1","addr":"172.21.15.87:6817","nonce":1086837511}]},"public_addr":"172.21.15.87:6805/1086837511","cluster_addr":"172.21.15.87:6812/1086837511","heartbeat_back_addr":"172.21.15.87:6822/1086837511","heartbeat_front_addr":"172.21.15.87:6817/1086837511","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.017629+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.043425+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T11:08:52.015849+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T11:51:44.183 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2023-12-16T11:51:44.183 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-16T11:51:44.318 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2023-12-16T11:51:44.319 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T11:51:44.319 INFO:teuthology.orchestra.run.smithi087.stdout:} 2023-12-16T11:51:44.329 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2023-12-16T11:51:44.607 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 0 to deep-scrub 2023-12-16T11:51:44.619 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2023-12-16T11:51:44.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-16T11:51:44.741 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2023-12-16T11:51:44.742 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T11:51:44.742 INFO:teuthology.orchestra.run.smithi087.stdout:} 2023-12-16T11:51:44.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2023-12-16T11:51:45.029 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 1 to deep-scrub 2023-12-16T11:51:45.042 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2023-12-16T11:51:45.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-16T11:51:45.165 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2023-12-16T11:51:45.165 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T11:51:45.165 INFO:teuthology.orchestra.run.smithi087.stdout:} 2023-12-16T11:51:45.175 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2023-12-16T11:51:45.454 INFO:teuthology.orchestra.run.smithi087.stderr:instructed osd(s) 2 to deep-scrub 2023-12-16T11:51:45.467 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-16T11:51:45.743 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:51:45.743 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-16T11:51:45.755 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1742,"stamp":"2023-12-16T11:51:44.872468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312416,"kb_used_data":12220,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282803104,"statfs":{"total":289910292480,"available":289590378496,"internally_reserved":0,"allocated":12513280,"data_stored":7270105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":307359055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002248"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:06.077280+0000","last_change":"2023-12-16T11:09:03.005273+0000","last_active":"2023-12-16T11:48:06.077280+0000","last_peered":"2023-12-16T11:48:06.077280+0000","last_clean":"2023-12-16T11:48:06.077280+0000","last_became_active":"2023-12-16T11:09:03.004893+0000","last_became_peered":"2023-12-16T11:09:03.004893+0000","last_unstale":"2023-12-16T11:48:06.077280+0000","last_undegraded":"2023-12-16T11:48:06.077280+0000","last_fullsized":"2023-12-16T11:48:06.077280+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:41:43.307235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:01.076765+0000","last_change":"2023-12-16T11:09:03.005695+0000","last_active":"2023-12-16T11:48:01.076765+0000","last_peered":"2023-12-16T11:48:01.076765+0000","last_clean":"2023-12-16T11:48:01.076765+0000","last_became_active":"2023-12-16T11:09:03.005445+0000","last_became_peered":"2023-12-16T11:09:03.005445+0000","last_unstale":"2023-12-16T11:48:01.076765+0000","last_undegraded":"2023-12-16T11:48:01.076765+0000","last_fullsized":"2023-12-16T11:48:01.076765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:17:51.300986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:51.082529+0000","last_change":"2023-12-16T11:09:03.005110+0000","last_active":"2023-12-16T11:48:51.082529+0000","last_peered":"2023-12-16T11:48:51.082529+0000","last_clean":"2023-12-16T11:48:51.082529+0000","last_became_active":"2023-12-16T11:09:03.004818+0000","last_became_peered":"2023-12-16T11:09:03.004818+0000","last_unstale":"2023-12-16T11:48:51.082529+0000","last_undegraded":"2023-12-16T11:48:51.082529+0000","last_fullsized":"2023-12-16T11:48:51.082529+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:13:30.010815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081386+0000","last_change":"2023-12-16T11:09:03.004551+0000","last_active":"2023-12-16T11:48:41.081386+0000","last_peered":"2023-12-16T11:48:41.081386+0000","last_clean":"2023-12-16T11:48:41.081386+0000","last_became_active":"2023-12-16T11:09:03.004235+0000","last_became_peered":"2023-12-16T11:09:03.004235+0000","last_unstale":"2023-12-16T11:48:41.081386+0000","last_undegraded":"2023-12-16T11:48:41.081386+0000","last_fullsized":"2023-12-16T11:48:41.081386+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:25:33.320035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":475,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:50:46.816934+0000","last_change":"2023-12-16T11:09:03.006609+0000","last_active":"2023-12-16T11:50:46.816934+0000","last_peered":"2023-12-16T11:50:46.816934+0000","last_clean":"2023-12-16T11:50:46.816934+0000","last_became_active":"2023-12-16T11:09:03.006307+0000","last_became_peered":"2023-12-16T11:09:03.006307+0000","last_unstale":"2023-12-16T11:50:46.816934+0000","last_undegraded":"2023-12-16T11:50:46.816934+0000","last_fullsized":"2023-12-16T11:50:46.816934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:36.265665+0000","last_change":"2023-12-16T11:09:03.004443+0000","last_active":"2023-12-16T11:48:36.265665+0000","last_peered":"2023-12-16T11:48:36.265665+0000","last_clean":"2023-12-16T11:48:36.265665+0000","last_became_active":"2023-12-16T11:09:03.004159+0000","last_became_peered":"2023-12-16T11:09:03.004159+0000","last_unstale":"2023-12-16T11:48:36.265665+0000","last_undegraded":"2023-12-16T11:48:36.265665+0000","last_fullsized":"2023-12-16T11:48:36.265665+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:10:30.195137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.269544+0000","last_change":"2023-12-16T11:09:03.004009+0000","last_active":"2023-12-16T11:49:11.269544+0000","last_peered":"2023-12-16T11:49:11.269544+0000","last_clean":"2023-12-16T11:49:11.269544+0000","last_became_active":"2023-12-16T11:09:03.003715+0000","last_became_peered":"2023-12-16T11:09:03.003715+0000","last_unstale":"2023-12-16T11:49:11.269544+0000","last_undegraded":"2023-12-16T11:49:11.269544+0000","last_fullsized":"2023-12-16T11:49:11.269544+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:12:11.654608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:48:41.081360+0000","last_change":"2023-12-16T11:09:03.006746+0000","last_active":"2023-12-16T11:48:41.081360+0000","last_peered":"2023-12-16T11:48:41.081360+0000","last_clean":"2023-12-16T11:48:41.081360+0000","last_became_active":"2023-12-16T11:09:03.006304+0000","last_became_peered":"2023-12-16T11:09:03.006304+0000","last_unstale":"2023-12-16T11:48:41.081360+0000","last_undegraded":"2023-12-16T11:48:41.081360+0000","last_fullsized":"2023-12-16T11:48:41.081360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:09:01.992011+0000","last_clean_scrub_stamp":"2023-12-16T11:09:01.992011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T16:20:34.752327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:49:11.084597+0000","last_change":"2023-12-16T11:08:58.062810+0000","last_active":"2023-12-16T11:49:11.084597+0000","last_peered":"2023-12-16T11:49:11.084597+0000","last_clean":"2023-12-16T11:49:11.084597+0000","last_became_active":"2023-12-16T11:08:58.061796+0000","last_became_peered":"2023-12-16T11:08:58.061796+0000","last_unstale":"2023-12-16T11:49:11.084597+0000","last_undegraded":"2023-12-16T11:49:11.084597+0000","last_fullsized":"2023-12-16T11:49:11.084597+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:08:56.977736+0000","last_clean_scrub_stamp":"2023-12-16T11:08:56.977736+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:33:16.597547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97268,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274572,"statfs":{"total":96636764160,"available":96537161728,"internally_reserved":0,"allocated":3776512,"data_stored":2030046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.375,"5min":0.50800000000000001,"15min":0.95699999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.19700000000000001},"max":{"1min":0.42699999999999999,"5min":0.54200000000000004,"15min":0.80800000000000005},"last":0.39800000000000002}]},{"osd":1,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.17899999999999999},"max":{"1min":0.42799999999999999,"5min":0.52500000000000002,"15min":0.626},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.35499999999999998,"15min":0.33600000000000002},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.55700000000000005,"15min":0.90900000000000003},"last":0.45800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738885,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93688,"kb_used_data":4268,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278152,"statfs":{"total":96636764160,"available":96540827648,"internally_reserved":0,"allocated":4370432,"data_stored":2620414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32400000000000001,"15min":0.313},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.184},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.45300000000000001,"5min":0.51000000000000001,"15min":0.58699999999999997},"last":0.23400000000000001}]},{"osd":2,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":0.504,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.496,"15min":0.51500000000000001},"last":0.29399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121460,"kb_used_data":4264,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250380,"statfs":{"total":96636764160,"available":96512389120,"internally_reserved":0,"allocated":4366336,"data_stored":2619645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.312,"15min":0.30599999999999999},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.16900000000000001},"max":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38300000000000001,"15min":0.34699999999999998},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.183},"max":{"1min":0.48999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.222},"max":{"1min":0.45300000000000001,"5min":0.49099999999999999,"15min":0.51800000000000002},"last":0.246},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42299999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.22600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:51:45.759 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.759 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.759 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.759 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.759 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.760 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.760 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.760 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-16T11:09:01.992011+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=9, tm_sec=1, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.760 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-16T11:08:56.977736+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=11, tm_min=51, tm_sec=43, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T11:51:45.760 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-16T11:52:05.761 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-16T11:52:06.048 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-12-16T11:52:06.048 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-12-16T11:52:06.060 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1752,"stamp":"2023-12-16T11:52:04.876409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312404,"kb_used_data":12220,"kb_used_omap":4,"kb_used_meta":300155,"kb_avail":282803116,"statfs":{"total":289910292480,"available":289590390784,"internally_reserved":0,"allocated":12513280,"data_stored":7270105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":307359055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002388"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":484,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:50.019860+0000","last_change":"2023-12-16T11:51:50.019884+0000","last_active":"2023-12-16T11:51:50.019860+0000","last_peered":"2023-12-16T11:51:50.019860+0000","last_clean":"2023-12-16T11:51:50.019860+0000","last_became_active":"2023-12-16T11:09:03.004893+0000","last_became_peered":"2023-12-16T11:09:03.004893+0000","last_unstale":"2023-12-16T11:51:50.019860+0000","last_undegraded":"2023-12-16T11:51:50.019860+0000","last_fullsized":"2023-12-16T11:51:50.019860+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:50.019752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:50.019752+0000","last_clean_scrub_stamp":"2023-12-16T11:51:50.019752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T19:12:56.487072+0000","scrub_duration":0.020503322000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":484,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:56.959345+0000","last_change":"2023-12-16T11:51:56.959369+0000","last_active":"2023-12-16T11:51:56.959345+0000","last_peered":"2023-12-16T11:51:56.959345+0000","last_clean":"2023-12-16T11:51:56.959345+0000","last_became_active":"2023-12-16T11:09:03.005445+0000","last_became_peered":"2023-12-16T11:09:03.005445+0000","last_unstale":"2023-12-16T11:51:56.959345+0000","last_undegraded":"2023-12-16T11:51:56.959345+0000","last_fullsized":"2023-12-16T11:51:56.959345+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:56.959205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:56.959205+0000","last_clean_scrub_stamp":"2023-12-16T11:51:56.959205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:40:08.977250+0000","scrub_duration":0.008016558,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":485,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:47.983053+0000","last_change":"2023-12-16T11:51:47.983075+0000","last_active":"2023-12-16T11:51:47.983053+0000","last_peered":"2023-12-16T11:51:47.983053+0000","last_clean":"2023-12-16T11:51:47.983053+0000","last_became_active":"2023-12-16T11:09:03.004818+0000","last_became_peered":"2023-12-16T11:09:03.004818+0000","last_unstale":"2023-12-16T11:51:47.983053+0000","last_undegraded":"2023-12-16T11:51:47.983053+0000","last_fullsized":"2023-12-16T11:51:47.983053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:47.982949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:47.982949+0000","last_clean_scrub_stamp":"2023-12-16T11:51:47.982949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T13:36:08.554466+0000","scrub_duration":0.017810683000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":485,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:48.958078+0000","last_change":"2023-12-16T11:51:48.958108+0000","last_active":"2023-12-16T11:51:48.958078+0000","last_peered":"2023-12-16T11:51:48.958078+0000","last_clean":"2023-12-16T11:51:48.958078+0000","last_became_active":"2023-12-16T11:09:03.004235+0000","last_became_peered":"2023-12-16T11:09:03.004235+0000","last_unstale":"2023-12-16T11:51:48.958078+0000","last_undegraded":"2023-12-16T11:51:48.958078+0000","last_fullsized":"2023-12-16T11:51:48.958078+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:48.957985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:48.957985+0000","last_clean_scrub_stamp":"2023-12-16T11:51:48.957985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:57:24.217859+0000","scrub_duration":0.0018121299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":483,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:45.239134+0000","last_change":"2023-12-16T11:51:45.239168+0000","last_active":"2023-12-16T11:51:45.239134+0000","last_peered":"2023-12-16T11:51:45.239134+0000","last_clean":"2023-12-16T11:51:45.239134+0000","last_became_active":"2023-12-16T11:09:03.006307+0000","last_became_peered":"2023-12-16T11:09:03.006307+0000","last_unstale":"2023-12-16T11:51:45.239134+0000","last_undegraded":"2023-12-16T11:51:45.239134+0000","last_fullsized":"2023-12-16T11:51:45.239134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:45.239018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:45.239018+0000","last_clean_scrub_stamp":"2023-12-16T11:51:45.239018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T20:29:03.102888+0000","scrub_duration":0.0062491079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":485,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:50.429999+0000","last_change":"2023-12-16T11:51:50.430034+0000","last_active":"2023-12-16T11:51:50.429999+0000","last_peered":"2023-12-16T11:51:50.429999+0000","last_clean":"2023-12-16T11:51:50.429999+0000","last_became_active":"2023-12-16T11:09:03.004159+0000","last_became_peered":"2023-12-16T11:09:03.004159+0000","last_unstale":"2023-12-16T11:51:50.429999+0000","last_undegraded":"2023-12-16T11:51:50.429999+0000","last_fullsized":"2023-12-16T11:51:50.429999+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:50.429883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:50.429883+0000","last_clean_scrub_stamp":"2023-12-16T11:51:50.429883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T18:45:45.047694+0000","scrub_duration":0.019907055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":486,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:46.439198+0000","last_change":"2023-12-16T11:51:46.439217+0000","last_active":"2023-12-16T11:51:46.439198+0000","last_peered":"2023-12-16T11:51:46.439198+0000","last_clean":"2023-12-16T11:51:46.439198+0000","last_became_active":"2023-12-16T11:09:03.003715+0000","last_became_peered":"2023-12-16T11:09:03.003715+0000","last_unstale":"2023-12-16T11:51:46.439198+0000","last_undegraded":"2023-12-16T11:51:46.439198+0000","last_fullsized":"2023-12-16T11:51:46.439198+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:46.439097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:46.439097+0000","last_clean_scrub_stamp":"2023-12-16T11:51:46.439097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T11:56:23.724839+0000","scrub_duration":0.0081632920000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":485,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:55.989785+0000","last_change":"2023-12-16T11:51:55.989818+0000","last_active":"2023-12-16T11:51:55.989785+0000","last_peered":"2023-12-16T11:51:55.989785+0000","last_clean":"2023-12-16T11:51:55.989785+0000","last_became_active":"2023-12-16T11:09:03.006304+0000","last_became_peered":"2023-12-16T11:09:03.006304+0000","last_unstale":"2023-12-16T11:51:55.989785+0000","last_undegraded":"2023-12-16T11:51:55.989785+0000","last_fullsized":"2023-12-16T11:51:55.989785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T11:51:55.989667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T11:51:55.989667+0000","last_clean_scrub_stamp":"2023-12-16T11:51:55.989667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T12:08:35.453468+0000","scrub_duration":0.018447640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"459'99","reported_seq":618,"reported_epoch":459,"state":"active+clean","last_fresh":"2023-12-16T11:51:45.043777+0000","last_change":"2023-12-16T11:51:45.043803+0000","last_active":"2023-12-16T11:51:45.043777+0000","last_peered":"2023-12-16T11:51:45.043777+0000","last_clean":"2023-12-16T11:51:45.043777+0000","last_became_active":"2023-12-16T11:08:58.061796+0000","last_became_peered":"2023-12-16T11:08:58.061796+0000","last_unstale":"2023-12-16T11:51:45.043777+0000","last_undegraded":"2023-12-16T11:51:45.043777+0000","last_fullsized":"2023-12-16T11:51:45.043777+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'99","last_scrub_stamp":"2023-12-16T11:51:45.043712+0000","last_deep_scrub":"459'99","last_deep_scrub_stamp":"2023-12-16T11:51:45.043712+0000","last_clean_scrub_stamp":"2023-12-16T11:51:45.043712+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T18:19:04.353539+0000","scrub_duration":0.0042228220000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97264,"kb_used_data":3688,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":94274576,"statfs":{"total":96636764160,"available":96537165824,"internally_reserved":0,"allocated":3776512,"data_stored":2030046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"min":{"1min":0.22700000000000001,"5min":0.221,"15min":0.17299999999999999},"max":{"1min":0.375,"5min":0.50800000000000001,"15min":0.95699999999999996},"last":0.375},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.27300000000000002,"5min":0.27100000000000002,"15min":0.19700000000000001},"max":{"1min":0.42699999999999999,"5min":0.54200000000000004,"15min":0.80800000000000005},"last":0.41799999999999998}]},{"osd":1,"last update":"Sat Dec 16 11:51:23 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.17899999999999999},"max":{"1min":0.42799999999999999,"5min":0.52500000000000002,"15min":0.626},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.35499999999999998,"15min":0.33600000000000002},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.20399999999999999},"max":{"1min":0.48799999999999999,"5min":0.55700000000000005,"15min":0.90900000000000003},"last":0.44}]}]},{"osd":1,"up_from":8,"seq":34359738889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93684,"kb_used_data":4268,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278156,"statfs":{"total":96636764160,"available":96540831744,"internally_reserved":0,"allocated":4370432,"data_stored":2620414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32400000000000001,"15min":0.313},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.184},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34399999999999997,"15min":0.33600000000000002},"min":{"1min":0.24099999999999999,"5min":0.23400000000000001,"15min":0.184},"max":{"1min":0.45300000000000001,"5min":0.51000000000000001,"15min":0.58699999999999997},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Dec 16 11:51:35 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.22500000000000001},"max":{"1min":0.504,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.379},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.223},"max":{"1min":0.45400000000000001,"5min":0.496,"15min":0.51500000000000001},"last":0.41699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738891,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121456,"kb_used_data":4264,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250384,"statfs":{"total":96636764160,"available":96512393216,"internally_reserved":0,"allocated":4366336,"data_stored":2619645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.312,"15min":0.30599999999999999},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.16900000000000001},"max":{"1min":0.41399999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38300000000000001,"15min":0.34699999999999998},"min":{"1min":0.34200000000000003,"5min":0.28100000000000003,"15min":0.183},"max":{"1min":0.48999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.377}]},{"osd":2,"last update":"Sat Dec 16 11:51:20 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35199999999999998,"15min":0.34799999999999998},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.222},"max":{"1min":0.45300000000000001,"5min":0.49099999999999999,"15min":0.51800000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33000000000000002},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42299999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T11:52:06.061 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2023-12-16T11:52:06.354 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-16T11:52:06.355 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-16T11:52:06.355 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-16T11:52:06.355 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-12-16T11:52:06.355 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:52:06.406 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-12-16T11:52:06.406 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-12-16T11:52:06.407 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:52:06.461 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-12-16T11:52:06.462 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-12-16T11:52:06.462 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:52:06.504 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-12-16T11:52:06.504 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-16T11:52:06.504 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-12-16T11:52:06.504 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:52:06.529 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-12-16T11:52:06.529 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-16T11:52:06.530 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-12-16T11:52:06.530 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T11:52:06.582 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-12-16T11:52:06.582 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-16T11:52:06.583 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-16T11:52:06.651 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:52:06.652 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-12-16T11:52:06.800 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:52:06.800 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-12-16T11:52:06.919 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:52:06.919 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-12-16T11:52:06.988 INFO:tasks.ceph:Archiving mon data... 2023-12-16T11:52:06.989 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/data/cluster1.mon.a.tgz 2023-12-16T11:52:06.989 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2023-12-16T11:52:07.004 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.pQ583aktTr 2023-12-16T11:52:07.004 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.pQ583aktTr 2023-12-16T11:52:07.138 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.pQ583aktTr 2023-12-16T11:52:07.222 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.pQ583aktTr is 281KB 2023-12-16T11:52:07.318 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.pQ583aktTr 2023-12-16T11:52:07.333 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-16T11:52:07.333 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-16T11:52:07.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-16T11:52:07.454 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-16T11:52:07.472 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-16T11:52:07.472 INFO:tasks.ceph:Archiving crash dumps... 2023-12-16T11:52:07.474 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/remote/smithi087/crash 2023-12-16T11:52:07.475 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-16T11:52:07.521 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/remote/smithi160/crash 2023-12-16T11:52:07.522 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-16T11:52:07.604 INFO:tasks.ceph:Compressing logs... 2023-12-16T11:52:07.604 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T11:52:07.606 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T11:52:07.630 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89956.log 2023-12-16T11:52:07.631 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-12-16T11:52:07.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89956.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89956.log.gz 2023-12-16T11:52:07.650 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-12-16T11:52:07.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89746.log 2023-12-16T11:52:07.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92321.log 2023-12-16T11:52:07.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89869.log 2023-12-16T11:52:07.650 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89746.log.gz 2023-12-16T11:52:07.650 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89620.log 2023-12-16T11:52:07.650 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92321.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89869.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89869.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90250.log 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89620.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89620.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.90250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90250.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89788.log 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89995.log 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89788.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92739.log 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92657.log 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89995.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89995.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92237.log 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92739.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92657.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92657.log.gz 2023-12-16T11:52:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84925.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92237.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92237.log.gz 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.84925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90208.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84925.log.gz 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89704.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.90208.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90208.log.gz 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92531.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89704.log.gz 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92279.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92615.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92531.log.gz 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92279.log.gz 2023-12-16T11:52:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90132.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92615.log.gz 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90082.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.90132.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90132.log.gz 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.89564.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.90082.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90082.log.gz 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84987.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89564.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89564.log.gz 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr: /var/log/ceph/cluster1-client.admin.92489.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.84987.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84987.log.gz 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92489.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92489.log.gz 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr: -- /var/log/ceph/cluster1-client.admin.84786.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84588.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.84786.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92363.log 2023-12-16T11:52:07.653 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.84786.log.gz 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90292.log 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.84588.log: /var/log/ceph/cluster1-client.admin.92363.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84588.log.gz 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92363.log.gz 88.1% -- replaced with /var/log/ceph/cluster1.log.gz 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89868.log 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89662.log 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.90292.log: /var/log/ceph/cluster1-client.admin.89868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90292.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92447.log 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89868.log.gz 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89662.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89662.log.gz 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89870.log 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92699.log 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92447.log: /var/log/ceph/cluster1-client.admin.89870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84826.log 2023-12-16T11:52:07.654 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92447.log.gz 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89870.log.gz 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92699.log.gz 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89996.log 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.84826.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84826.log.gz 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80120.log 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89996.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89996.log.gz 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.tmp-client.admin.80120.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80120.log.gz 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90102.log 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84746.log 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.90102.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89453.log 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.90102.log.gz 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.84746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92573.log 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.89453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.84746.log.gz 2023-12-16T11:52:07.655 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89453.log.gz 2023-12-16T11:52:07.656 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92405.log 2023-12-16T11:52:07.656 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92573.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92573.log.gz 2023-12-16T11:52:07.656 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.92405.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92405.log.gz 2023-12-16T11:52:07.668 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98785.log 2023-12-16T11:52:07.668 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107683.log 2023-12-16T11:52:07.669 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139701.log 2023-12-16T11:52:07.669 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104486.log 2023-12-16T11:52:07.669 INFO:teuthology.orchestra.run.smithi160.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98785.log.gz 2023-12-16T11:52:07.669 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103273.log 2023-12-16T11:52:07.669 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139701.log.gz 2023-12-16T11:52:07.669 INFO:teuthology.orchestra.run.smithi160.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107683.log.gz 2023-12-16T11:52:07.669 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.104486.log: -- /var/log/ceph/cluster2-client.mirror.102275.log 2023-12-16T11:52:07.677 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120717.log 2023-12-16T11:52:07.677 INFO:teuthology.orchestra.run.smithi160.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103273.log.gz 2023-12-16T11:52:07.677 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105623.log 2023-12-16T11:52:07.677 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102275.log.gz 2023-12-16T11:52:07.677 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132936.log 2023-12-16T11:52:07.677 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120717.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115367.log 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132936.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105623.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109081.log 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132936.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97639.log 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108818.log 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131508.log 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109081.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97639.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135753.log 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115367.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101360.log 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131508.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131508.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135753.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135753.log.gz 2023-12-16T11:52:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113895.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132180.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101360.log: /var/log/ceph/cluster2-client.mirror.113895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129788.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4%/var/log/ceph/cluster2-client.admin.132180.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108818.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.113895.log.gz 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132180.log.gz 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120152.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101360.log.gz 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117002.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.129788.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129788.log.gz 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98441.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109211.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120152.log.gz 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101380.log 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117002.log.gz 2023-12-16T11:52:07.679 INFO:teuthology.orchestra.run.smithi160.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.98441.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109211.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114934.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.101380.log: -- replaced with /var/log/ceph/cluster1-client.mirror.109211.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101131.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101380.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133986.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107727.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114934.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133986.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133986.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122630.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136047.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107727.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99691.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122630.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133650.log 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136047.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136047.log.gz 2023-12-16T11:52:07.680 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121572.log 2023-12-16T11:52:07.681 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133650.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99691.log.gz 0.0% 2023-12-16T11:52:07.681 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.133650.log.gz 2023-12-16T11:52:07.681 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98717.log 2023-12-16T11:52:07.681 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124776.log 2023-12-16T11:52:07.721 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121572.log: /var/log/ceph/cluster2-client.mirror.98717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118076.log 2023-12-16T11:52:07.721 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121572.log.gz 2023-12-16T11:52:07.721 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.124776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120426.log 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98717.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124776.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122225.log 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92059.log 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120426.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120426.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94262.log 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122225.log: /var/log/ceph/cluster1-client.admin.92059.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122225.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132222.log 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.92059.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95223.log 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118076.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132222.log: gzip -5 88.4% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.94262.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr: /var/log/ceph/cluster1-client.mirror.125610.log 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132222.log.gz 2023-12-16T11:52:07.722 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126293.log 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.95223.log.gz 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121730.log 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125610.log.gz 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102325.log 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126293.log: /var/log/ceph/cluster1-client.mirror.121730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105557.log 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126293.log.gz 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121730.log.gz 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105647.log 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102325.log.gz 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125822.log 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105557.log: /var/log/ceph/cluster1-client.mirror.105647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135795.log 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125822.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105647.log.gz 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose 88.4% -- /var/log/ceph/cluster2-client.mirror.95407.log -- replaced with /var/log/ceph/cluster2-client.mirror.105557.log.gz 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.723 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.100567.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135795.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125822.log.gz 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135795.log.gz 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116263.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.0.100567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115076.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111768.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95012.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111768.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115076.log.gz 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96686.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111768.log.gz 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.91684.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr: 92.3%/var/log/ceph/cluster2-client.mirror.95012.log: /var/log/ceph/cluster1-client.admin.96686.log: -- replaced with /var/log/ceph/cluster1-client.mirror.116263.log.gz 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110681.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.0.91684.log: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117206.log 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.96686.log.gz 2023-12-16T11:52:07.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127539.log 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117206.log: 89.0% 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.100567.log.gz 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.110681.log.gz 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117206.log.gz 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139239.log 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.127539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124883.log 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95407.log.gz 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9%/var/log/ceph/cluster2-client.admin.139239.log: -- replaced with /var/log/ceph/cluster1-client.mirror.127539.log.gz 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139239.log.gz 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125698.log 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.124883.log: --verbose -- /var/log/ceph/cluster1-client.mirror.114032.log 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125698.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124883.log.gz 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93457.log 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125698.log.gz 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94094.log 2023-12-16T11:52:07.725 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.93457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132054.log 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.94094.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.93457.log.gz 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110275.log 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.94094.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108480.log 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132054.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132054.log.gz 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:gzip 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95012.log.gz 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.135711.log 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110275.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114032.log.gz 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107276.log 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135711.log: 87.4% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110275.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.108480.log.gz 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117952.log 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135711.log.gz 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106481.log 2023-12-16T11:52:07.726 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107276.log.gz 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.97273.log 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118312.log 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106481.log: /var/log/ceph/cluster2-client.mirror.1.97273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110725.log 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115121.log 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117952.log.gz 87.6% 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118312.log.gz 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108860.log 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106481.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110725.log.gz 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115121.log: 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97509.log 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100656.log 2023-12-16T11:52:07.727 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108860.log: /var/log/ceph/cluster1-client.mirror.97509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90860.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100656.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97509.log.gz 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100656.log.gz 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108860.log.gz 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:gzip 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115121.log.gz 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113184.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90860.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90860.log.gz 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126272.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107840.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126272.log: 88.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99003.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.113184.log.gz 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126272.log.gz 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95032.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133860.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129129.log 2023-12-16T11:52:07.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133860.log: 92.4% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133860.log.gz 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107840.log.gz 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129406.log 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.129129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95780.log 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129129.log.gz 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129406.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129406.log.gz 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr: /var/log/ceph/cluster1-client.mirror.111858.log 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98831.log 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111858.log: /var/log/ceph/cluster2-client.mirror.95780.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111858.log.gz 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92750.log 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98831.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99003.log.gz 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.91684.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96970.log 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95032.log.gz 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129152.log 2023-12-16T11:52:07.729 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96970.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92750.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139533.log 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.129152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116038.log 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.96970.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139533.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129152.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109748.log 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116038.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139533.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139743.log 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111523.log 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4%/var/log/ceph/cluster2-client.admin.139743.log: -- replaced with /var/log/ceph/cluster2-client.mirror.109748.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139743.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104523.log 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr: 93.3% 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116038.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.95780.log.gz 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85991.log/var/log/ceph/cluster2-client.mirror.111523.log: 2023-12-16T11:52:07.730 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104523.log: gzip -5 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98831.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.112433.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111523.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.85991.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85991.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102737.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104523.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133818.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120515.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133818.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111834.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.133818.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102737.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109276.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120515.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112433.log.gz 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127727.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111834.log: /var/log/ceph/cluster1-client.mirror.109276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100264.log 2023-12-16T11:52:07.731 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111834.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138441.log 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.100264.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127727.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121325.log 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100264.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103734.log 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138441.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109276.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117495.log 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121325.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101497.log 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135249.log 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101497.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117495.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr: -- /var/log/ceph/cluster1-client.mirror.113943.log 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101497.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135249.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135249.log.gz 2023-12-16T11:52:07.732 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108376.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127850.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113943.log.gz 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137685.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130794.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127850.log.gz 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137685.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137685.log.gz 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119615.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130794.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120069.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.130794.log.gz 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108376.log.gz 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139029.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135333.log 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119615.log.gz 2023-12-16T11:52:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120069.log: /var/log/ceph/cluster2-client.admin.139029.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.139029.log.gz 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97767.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92011.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135333.log.gz 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134283.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.92011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99195.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.92011.log.gz 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134283.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134283.log.gz 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93433.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120069.log.gz 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117288.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121685.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117288.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117288.log.gz 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102799.log 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121685.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121685.log.gz 2023-12-16T11:52:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97146.log 2023-12-16T11:52:07.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102799.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102799.log.gz 2023-12-16T11:52:07.773 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94308.log 2023-12-16T11:52:07.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97146.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99195.log.gz 2023-12-16T11:52:07.773 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121145.log 2023-12-16T11:52:07.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94308.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94308.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126585.log 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121145.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121145.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111412.log 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126585.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126585.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112135.log 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111412.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111412.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99152.log 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112135.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112135.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139449.log 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99152.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99152.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129344.log 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139449.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105015.log 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129344.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129344.log.gz 2023-12-16T11:52:07.774 INFO:teuthology.orchestra.run.smithi160.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.97273.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104169.log 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122427.log 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104169.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105015.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104169.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92035.log 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126231.log 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122427.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.92035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125802.log 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0%/var/log/ceph/cluster2-client.mirror.126231.log: -- replaced with /var/log/ceph/cluster1-client.admin.92035.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:gzip 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126231.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.90774.log 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109432.log 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90774.log: /var/log/ceph/cluster2-client.mirror.125802.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90774.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125802.log.gz 2023-12-16T11:52:07.775 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121302.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126811.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109432.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121302.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121302.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124862.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105386.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126811.log: /var/log/ceph/cluster2-client.mirror.124862.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124862.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121167.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105386.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105386.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126811.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119429.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129812.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121167.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115100.log 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.129812.log: 87.8% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.129812.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119429.log.gz 2023-12-16T11:52:07.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102342.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102090.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115100.log.gz 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118374.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102090.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102342.log.gz 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102090.log.gz 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128991.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121842.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118374.log.gz 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128991.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128991.log.gz 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126770.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136593.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121842.log.gz 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116777.log 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136593.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136593.log.gz 2023-12-16T11:52:07.777 INFO:teuthology.orchestra.run.smithi160.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126770.log.gz 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113919.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120537.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113919.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116777.log.gz 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113919.log.gz 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132642.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116592.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120537.log.gz 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91092.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132642.log.gz/var/log/ceph/cluster1-client.mirror.116592.log: 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96828.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116592.log.gz 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117842.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.91092.log: /var/log/ceph/cluster2-client.mirror.96828.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91092.log.gz 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101594.log 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96828.log.gz 2023-12-16T11:52:07.778 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124985.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101594.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101594.log.gz 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111681.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124985.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124985.log.gz 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117842.log.gz 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131886.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129467.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111681.log.gz 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131886.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131886.log.gz 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129323.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125921.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129467.log.gz 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.129323.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.129323.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95130.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108078.log 2023-12-16T11:52:07.779 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125921.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121437.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108078.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95130.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108078.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112585.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113405.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121437.log: /var/log/ceph/cluster1-client.mirror.112585.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121437.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110990.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112585.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113405.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125006.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110990.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110990.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113405.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118229.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116351.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125006.log.gz 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117722.log 2023-12-16T11:52:07.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116351.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118229.log.gz 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97811.log 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116351.log.gz 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101323.log 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117722.log: /var/log/ceph/cluster1-client.admin.97811.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.97811.log.gz 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120807.log 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101323.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117722.log.gz 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102878.log 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120807.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120807.log.gz 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114261.log 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102878.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102878.log.gz 2023-12-16T11:52:07.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100751.log 2023-12-16T11:52:07.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114261.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114261.log.gz 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126520.log 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100751.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100751.log.gz 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100788.log 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126520.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126520.log.gz 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125720.log 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100788.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100788.log.gz 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109996.log 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125720.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125720.log.gz 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122793.log 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109996.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109996.log.gz 2023-12-16T11:52:07.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111705.log 2023-12-16T11:52:07.793 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.122793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116685.log 2023-12-16T11:52:07.794 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111705.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111705.log.gz 2023-12-16T11:52:07.808 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111906.log 2023-12-16T11:52:07.818 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116685.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112963.log 2023-12-16T11:52:07.818 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111906.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111906.log.gz 2023-12-16T11:52:07.822 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116685.log.gz 2023-12-16T11:52:07.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128786.log 2023-12-16T11:52:07.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112963.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112963.log.gz 2023-12-16T11:52:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90420.log 2023-12-16T11:52:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128786.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128786.log.gz 2023-12-16T11:52:07.853 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119160.log 2023-12-16T11:52:07.853 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90420.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90420.log.gz 2023-12-16T11:52:07.862 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102816.log 2023-12-16T11:52:07.862 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119160.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119160.log.gz 2023-12-16T11:52:07.872 INFO:teuthology.orchestra.run.smithi087.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-12-16T11:52:07.879 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121932.log 2023-12-16T11:52:07.880 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102816.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102816.log.gz 2023-12-16T11:52:07.888 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112183.log 2023-12-16T11:52:07.898 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127583.log 2023-12-16T11:52:07.902 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112183.log: 87.8% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121932.log.gz 2023-12-16T11:52:07.904 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104486.log.gz 2023-12-16T11:52:07.908 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119491.log 2023-12-16T11:52:07.908 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127583.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127583.log.gz 2023-12-16T11:52:07.908 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105215.log 2023-12-16T11:52:07.908 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119491.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119491.log.gz 2023-12-16T11:52:07.908 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93836.log 2023-12-16T11:52:07.911 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105215.log: -- replaced with /var/log/ceph/cluster1-client.mirror.112183.log.gz 2023-12-16T11:52:07.918 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119015.log 2023-12-16T11:52:07.919 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93836.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93836.log.gz 2023-12-16T11:52:07.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113031.log 2023-12-16T11:52:07.936 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119015.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119015.log.gz 2023-12-16T11:52:07.949 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98761.log 2023-12-16T11:52:07.949 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113031.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113031.log.gz 2023-12-16T11:52:07.957 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118126.log 2023-12-16T11:52:07.963 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98761.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98761.log.gz 2023-12-16T11:52:07.963 INFO:teuthology.orchestra.run.smithi087.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-12-16T11:52:07.967 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126434.log 2023-12-16T11:52:07.976 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118126.log: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137895.log 2023-12-16T11:52:07.980 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126434.log: -- replaced with /var/log/ceph/cluster2-client.mirror.118126.log.gz 2023-12-16T11:52:07.985 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101323.log.gz 2023-12-16T11:52:07.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95319.log 2023-12-16T11:52:07.989 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137895.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137895.log.gz 2023-12-16T11:52:08.001 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108010.log 2023-12-16T11:52:08.002 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95319.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95319.log.gz 2023-12-16T11:52:08.015 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138189.log 2023-12-16T11:52:08.016 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108010.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108010.log.gz 2023-12-16T11:52:08.017 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97412.log 2023-12-16T11:52:08.017 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138189.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138189.log.gz 2023-12-16T11:52:08.028 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110635.log 2023-12-16T11:52:08.029 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97412.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.97412.log.gz 2023-12-16T11:52:08.031 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97767.log.gz 2023-12-16T11:52:08.032 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109589.log 2023-12-16T11:52:08.032 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110635.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110635.log.gz 2023-12-16T11:52:08.035 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90840.log 2023-12-16T11:52:08.035 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109589.log: 86.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109589.log.gz 2023-12-16T11:52:08.035 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108528.log 2023-12-16T11:52:08.035 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90840.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90840.log.gz 2023-12-16T11:52:08.035 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126849.log 2023-12-16T11:52:08.036 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108528.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108528.log.gz 2023-12-16T11:52:08.037 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106581.log 2023-12-16T11:52:08.037 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126849.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126849.log.gz 2023-12-16T11:52:08.038 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110049.log 2023-12-16T11:52:08.038 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106581.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106581.log.gz 2023-12-16T11:52:08.038 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121887.log 2023-12-16T11:52:08.040 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110049.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.110049.log.gz 2023-12-16T11:52:08.041 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104741.log 2023-12-16T11:52:08.041 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121887.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121887.log.gz 2023-12-16T11:52:08.042 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119201.log 2023-12-16T11:52:08.062 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104741.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97146.log.gz 2023-12-16T11:52:08.062 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92161.log 2023-12-16T11:52:08.065 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119201.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119201.log.gz 2023-12-16T11:52:08.065 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102836.log 2023-12-16T11:52:08.065 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.92161.log: 56.4% -- replaced with /var/log/ceph/cluster1-client.mirror.92161.log.gz 2023-12-16T11:52:08.065 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114104.log 2023-12-16T11:52:08.068 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102836.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102836.log.gz 2023-12-16T11:52:08.068 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100306.log 2023-12-16T11:52:08.068 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114104.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114104.log.gz 2023-12-16T11:52:08.068 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92076.log 2023-12-16T11:52:08.069 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100306.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100306.log.gz 2023-12-16T11:52:08.069 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101131.log.gz 2023-12-16T11:52:08.069 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101281.log 2023-12-16T11:52:08.071 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.92076.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.92076.log.gz 2023-12-16T11:52:08.071 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116106.log 2023-12-16T11:52:08.071 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.101281.log: --verbose -- /var/log/ceph/cluster2-client.mirror.118623.log 2023-12-16T11:52:08.071 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116106.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101281.log.gz 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116106.log.gz 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104761.log 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96261.log 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.118623.log.gz 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138903.log 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96261.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104761.log.gz 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120049.log 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96261.log.gz 2023-12-16T11:52:08.076 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138903.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138903.log.gz 2023-12-16T11:52:08.077 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126354.log 2023-12-16T11:52:08.077 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.120049.log: -- /var/log/ceph/cluster2-client.admin.136425.log 2023-12-16T11:52:08.077 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120049.log.gz 2023-12-16T11:52:08.082 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126354.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126354.log.gz 2023-12-16T11:52:08.082 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132852.log 2023-12-16T11:52:08.082 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136425.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119077.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.136425.log.gz 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112609.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119077.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132852.log.gz 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100918.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112609.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112609.log.gz 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128208.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119077.log.gz 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111215.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128208.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100918.log.gz 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119036.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128208.log.gz 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111215.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111215.log.gz 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101614.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108902.log 2023-12-16T11:52:08.083 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119036.log: /var/log/ceph/cluster2-client.mirror.101614.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119036.log.gz 2023-12-16T11:52:08.084 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101614.log.gz 2023-12-16T11:52:08.084 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129836.log 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122090.log 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108902.log: /var/log/ceph/cluster2-client.admin.129836.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129836.log.gz 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108902.log.gz 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121347.log 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97917.log 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122090.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103734.log.gz 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121347.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122090.log.gz 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121347.log.gz 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127768.log 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90123.log 2023-12-16T11:52:08.086 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97917.log: /var/log/ceph/cluster2-client.mirror.127768.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125761.log 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90123.log: 87.8% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90123.log.gz 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.127768.log.gz 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97917.log.gz 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137643.log 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.125761.log: -- /var/log/ceph/cluster1-client.mirror.116616.log 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137643.log: 87.7% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137643.log.gz 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114630.log 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.125761.log.gz 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114413.log 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137475.log 2023-12-16T11:52:08.090 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116616.log.gz/var/log/ceph/cluster1-client.mirror.114630.log: 2023-12-16T11:52:08.091 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114413.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114630.log.gz 2023-12-16T11:52:08.091 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114413.log.gz 2023-12-16T11:52:08.091 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119449.log 2023-12-16T11:52:08.091 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107775.log 2023-12-16T11:52:08.091 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137475.log: /var/log/ceph/cluster2-client.mirror.119449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102937.log 2023-12-16T11:52:08.091 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137475.log.gz 2023-12-16T11:52:08.091 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107775.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107775.log.gz 2023-12-16T11:52:08.096 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108597.log 2023-12-16T11:52:08.096 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102937.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119449.log.gz 2023-12-16T11:52:08.096 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129426.log 2023-12-16T11:52:08.096 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106523.log 2023-12-16T11:52:08.096 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129426.log: /var/log/ceph/cluster2-client.mirror.108597.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129426.log.gz 2023-12-16T11:52:08.096 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109569.log 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106523.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106523.log.gz 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108597.log.gz 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121122.log 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109340.log 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121122.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109569.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121122.log.gz 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110410.log 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95639.log 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109340.log: /var/log/ceph/cluster2-client.mirror.110410.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109340.log.gz 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110410.log.gz 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94489.log 2023-12-16T11:52:08.097 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102641.log 2023-12-16T11:52:08.099 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95639.log: /var/log/ceph/cluster1-client.mirror.94489.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95639.log.gz 2023-12-16T11:52:08.099 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103332.log 2023-12-16T11:52:08.099 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102641.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102641.log.gz 2023-12-16T11:52:08.099 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124440.log 2023-12-16T11:52:08.103 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103332.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.94489.log.gz 2023-12-16T11:52:08.103 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97082.log 2023-12-16T11:52:08.104 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124440.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124440.log.gz 2023-12-16T11:52:08.104 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113449.log 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97082.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122793.log.gz 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97082.log.gz 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134829.log 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131550.log 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134829.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113449.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134829.log.gz 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140037.log 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129529.log 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131550.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131550.log.gz 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140037.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140037.log.gz 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101636.log 2023-12-16T11:52:08.112 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129529.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129529.log.gz 2023-12-16T11:52:08.113 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98098.log 2023-12-16T11:52:08.113 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102957.log 2023-12-16T11:52:08.119 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101636.log: /var/log/ceph/cluster1-client.mirror.98098.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98098.log.gz 2023-12-16T11:52:08.119 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127829.log 2023-12-16T11:52:08.119 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102957.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102957.log.gz 2023-12-16T11:52:08.120 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125026.log 2023-12-16T11:52:08.122 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127829.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127829.log.gz 2023-12-16T11:52:08.122 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122562.log 2023-12-16T11:52:08.122 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125026.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125026.log.gz 2023-12-16T11:52:08.128 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116375.log 2023-12-16T11:52:08.129 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122562.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122562.log.gz 2023-12-16T11:52:08.129 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120090.log 2023-12-16T11:52:08.129 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116375.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116375.log.gz 2023-12-16T11:52:08.129 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101402.log 2023-12-16T11:52:08.129 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120090.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120090.log.gz 2023-12-16T11:52:08.129 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99218.log 2023-12-16T11:52:08.138 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111882.log 2023-12-16T11:52:08.144 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99218.log: 93.4% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.93433.log.gz 2023-12-16T11:52:08.152 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114654.log 2023-12-16T11:52:08.152 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111882.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111882.log.gz 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.99218.log.gz 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130080.log 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114654.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114654.log.gz 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118498.log 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130080.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130080.log.gz 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108126.log 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118498.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118498.log.gz 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107411.log 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108126.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108126.log.gz 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116198.log 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107411.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107411.log.gz 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120255.log 2023-12-16T11:52:08.153 INFO:teuthology.orchestra.run.smithi160.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126434.log.gz 2023-12-16T11:52:08.159 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116198.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116198.log.gz 2023-12-16T11:52:08.159 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91510.log 2023-12-16T11:52:08.159 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101710.log 2023-12-16T11:52:08.159 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.91510.log: 83.2% -- replaced with /var/log/ceph/cluster2-client.mirror.91510.log.gz 2023-12-16T11:52:08.159 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114321.log 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101710.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120255.log.gz 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101710.log.gz 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125999.log 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85617.log 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125999.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114321.log.gz 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125999.log.gz 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.91604.log 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.85617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85617.log.gz 2023-12-16T11:52:08.160 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138861.log 2023-12-16T11:52:08.161 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.2.91604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113321.log 2023-12-16T11:52:08.161 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138861.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138861.log.gz 2023-12-16T11:52:08.161 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119222.log 2023-12-16T11:52:08.162 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113321.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113321.log.gz 2023-12-16T11:52:08.162 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129173.log 2023-12-16T11:52:08.162 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119222.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119222.log.gz 2023-12-16T11:52:08.162 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115997.log 2023-12-16T11:52:08.162 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129173.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129173.log.gz 2023-12-16T11:52:08.163 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99046.log 2023-12-16T11:52:08.164 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115997.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115997.log.gz 2023-12-16T11:52:08.165 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125214.log 2023-12-16T11:52:08.169 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99046.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99046.log.gz 2023-12-16T11:52:08.170 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103771.log 2023-12-16T11:52:08.171 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125214.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125214.log.gz 2023-12-16T11:52:08.172 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105740.log 2023-12-16T11:52:08.175 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.103771.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.103771.log.gz 2023-12-16T11:52:08.175 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101264.log 2023-12-16T11:52:08.177 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105740.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105740.log.gz 2023-12-16T11:52:08.178 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130668.log 2023-12-16T11:52:08.179 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101264.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101264.log.gz 2023-12-16T11:52:08.179 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102658.log 2023-12-16T11:52:08.179 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130668.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130668.log.gz 2023-12-16T11:52:08.179 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106957.log 2023-12-16T11:52:08.180 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102658.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102658.log.gz 2023-12-16T11:52:08.180 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95427.log 2023-12-16T11:52:08.181 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106957.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106957.log.gz 2023-12-16T11:52:08.182 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113341.log 2023-12-16T11:52:08.187 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95427.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95427.log.gz 2023-12-16T11:52:08.188 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137013.log 2023-12-16T11:52:08.189 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113341.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113341.log.gz 2023-12-16T11:52:08.190 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103613.log 2023-12-16T11:52:08.190 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137013.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137013.log.gz 2023-12-16T11:52:08.191 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127288.log 2023-12-16T11:52:08.191 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103613.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103613.log.gz 2023-12-16T11:52:08.193 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106740.log 2023-12-16T11:52:08.193 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127288.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127288.log.gz 2023-12-16T11:52:08.193 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122022.log 2023-12-16T11:52:08.193 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106740.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106740.log.gz 2023-12-16T11:52:08.194 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132978.log 2023-12-16T11:52:08.194 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122022.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122022.log.gz 2023-12-16T11:52:08.195 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122675.log 2023-12-16T11:52:08.195 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132978.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132978.log.gz 2023-12-16T11:52:08.196 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116423.log 2023-12-16T11:52:08.196 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122675.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122675.log.gz 2023-12-16T11:52:08.197 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106653.log 2023-12-16T11:52:08.197 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116423.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116423.log.gz 2023-12-16T11:52:08.198 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116510.log 2023-12-16T11:52:08.198 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106653.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106653.log.gz 2023-12-16T11:52:08.199 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117433.log 2023-12-16T11:52:08.199 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116510.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116510.log.gz 2023-12-16T11:52:08.200 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101974.log 2023-12-16T11:52:08.201 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117433.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117433.log.gz 2023-12-16T11:52:08.204 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126667.log 2023-12-16T11:52:08.204 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101974.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101974.log.gz 2023-12-16T11:52:08.204 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.91640.log 2023-12-16T11:52:08.204 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126667.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126667.log.gz 2023-12-16T11:52:08.204 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118415.log 2023-12-16T11:52:08.206 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.3.91640.log: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.91604.log.gz 2023-12-16T11:52:08.206 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107530.log 2023-12-16T11:52:08.209 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118415.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118415.log.gz 2023-12-16T11:52:08.209 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99496.log 2023-12-16T11:52:08.209 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107530.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107530.log.gz 2023-12-16T11:52:08.209 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111344.log 2023-12-16T11:52:08.215 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99496.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99496.log.gz 2023-12-16T11:52:08.215 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137433.log 2023-12-16T11:52:08.217 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111344.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111344.log.gz 2023-12-16T11:52:08.217 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119181.log 2023-12-16T11:52:08.217 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137433.log.gz 2023-12-16T11:52:08.217 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117371.log 2023-12-16T11:52:08.218 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119181.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119181.log.gz 2023-12-16T11:52:08.220 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139071.log 2023-12-16T11:52:08.220 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117371.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117371.log.gz 2023-12-16T11:52:08.220 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116243.log 2023-12-16T11:52:08.221 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139071.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139071.log.gz 2023-12-16T11:52:08.221 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98625.log 2023-12-16T11:52:08.224 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116243.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116243.log.gz 2023-12-16T11:52:08.225 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90588.log 2023-12-16T11:52:08.227 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98625.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98625.log.gz 2023-12-16T11:52:08.230 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110299.log 2023-12-16T11:52:08.230 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90588.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90588.log.gz 2023-12-16T11:52:08.230 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99175.log 2023-12-16T11:52:08.230 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110299.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110299.log.gz 2023-12-16T11:52:08.230 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.107899.log 2023-12-16T11:52:08.236 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99175.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99175.log.gz 2023-12-16T11:52:08.236 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110251.log 2023-12-16T11:52:08.246 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.1.107899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125657.log 2023-12-16T11:52:08.246 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110251.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110251.log.gz 2023-12-16T11:52:08.260 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94576.log 2023-12-16T11:52:08.260 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125657.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125657.log.gz 2023-12-16T11:52:08.270 INFO:teuthology.orchestra.run.smithi160.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.107899.log.gz 2023-12-16T11:52:08.270 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121910.log 2023-12-16T11:52:08.283 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125462.log 2023-12-16T11:52:08.283 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121910.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121910.log.gz 2023-12-16T11:52:08.292 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130332.log 2023-12-16T11:52:08.292 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125462.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125462.log.gz 2023-12-16T11:52:08.294 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105215.log.gz 2023-12-16T11:52:08.304 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99410.log 2023-12-16T11:52:08.305 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130332.log.gz 2023-12-16T11:52:08.305 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140331.log 2023-12-16T11:52:08.311 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99410.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99410.log.gz 2023-12-16T11:52:08.311 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116705.log 2023-12-16T11:52:08.312 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140331.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140331.log.gz 2023-12-16T11:52:08.312 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124643.log 2023-12-16T11:52:08.315 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116705.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116705.log.gz 2023-12-16T11:52:08.315 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103675.log 2023-12-16T11:52:08.316 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124643.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124643.log.gz 2023-12-16T11:52:08.316 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.91790.log 2023-12-16T11:52:08.317 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103675.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103675.log.gz 2023-12-16T11:52:08.317 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132264.log 2023-12-16T11:52:08.326 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.3.91790.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103332.log.gz 2023-12-16T11:52:08.328 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113714.log 2023-12-16T11:52:08.328 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132264.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132264.log.gz 2023-12-16T11:52:08.328 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94900.log 2023-12-16T11:52:08.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113714.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113714.log.gz 2023-12-16T11:52:08.331 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107599.log 2023-12-16T11:52:08.331 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94900.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94900.log.gz 2023-12-16T11:52:08.331 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99731.log 2023-12-16T11:52:08.332 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107599.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107599.log.gz 2023-12-16T11:52:08.333 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133020.log 2023-12-16T11:52:08.334 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99731.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99731.log.gz 2023-12-16T11:52:08.334 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126543.log 2023-12-16T11:52:08.338 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133020.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133020.log.gz 2023-12-16T11:52:08.338 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90294.log 2023-12-16T11:52:08.338 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126543.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126543.log.gz 2023-12-16T11:52:08.339 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120740.log 2023-12-16T11:52:08.339 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90294.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90294.log.gz 2023-12-16T11:52:08.339 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94352.log 2023-12-16T11:52:08.339 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120740.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120740.log.gz 2023-12-16T11:52:08.339 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124272.log 2023-12-16T11:52:08.340 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94352.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94352.log.gz 2023-12-16T11:52:08.340 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114871.log 2023-12-16T11:52:08.340 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.124272.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124272.log.gz 2023-12-16T11:52:08.342 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94146.log 2023-12-16T11:52:08.342 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114871.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114871.log.gz 2023-12-16T11:52:08.342 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102404.log 2023-12-16T11:52:08.343 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.94146.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94146.log.gz 2023-12-16T11:52:08.344 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.105253.log 2023-12-16T11:52:08.345 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102404.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102404.log.gz 2023-12-16T11:52:08.345 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119635.log 2023-12-16T11:52:08.348 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.2.105253.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.91790.log.gz 2023-12-16T11:52:08.348 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113991.log 2023-12-16T11:52:08.351 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119635.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119635.log.gz 2023-12-16T11:52:08.351 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.117842.log 2023-12-16T11:52:08.351 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113991.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113991.log.gz 2023-12-16T11:52:08.352 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98602.log 2023-12-16T11:52:08.365 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.117842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92144.log 2023-12-16T11:52:08.372 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98602.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127124.log 2023-12-16T11:52:08.372 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98602.log.gz 2023-12-16T11:52:08.379 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.92144.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.117842.log.gz 2023-12-16T11:52:08.380 INFO:teuthology.orchestra.run.smithi160.stderr: 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92144.log.gz 2023-12-16T11:52:08.380 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109456.log 2023-12-16T11:52:08.380 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127124.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127124.log.gz 2023-12-16T11:52:08.380 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116150.log 2023-12-16T11:52:08.380 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109456.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109456.log.gz 2023-12-16T11:52:08.381 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95573.log 2023-12-16T11:52:08.381 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116150.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.116150.log.gz 2023-12-16T11:52:08.382 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96712.log 2023-12-16T11:52:08.382 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95573.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95573.log.gz 2023-12-16T11:52:08.382 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99971.log 2023-12-16T11:52:08.383 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.96712.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.96712.log.gz 2023-12-16T11:52:08.383 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115789.log 2023-12-16T11:52:08.384 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99971.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99971.log.gz 2023-12-16T11:52:08.385 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136845.log 2023-12-16T11:52:08.386 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115789.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115789.log.gz 2023-12-16T11:52:08.387 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105671.log 2023-12-16T11:52:08.387 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136845.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136845.log.gz 2023-12-16T11:52:08.387 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100567.log 2023-12-16T11:52:08.387 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105671.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105671.log.gz 2023-12-16T11:52:08.389 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113204.log 2023-12-16T11:52:08.389 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.100567.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100567.log.gz 2023-12-16T11:52:08.389 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113120.log 2023-12-16T11:52:08.389 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113204.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113204.log.gz 2023-12-16T11:52:08.390 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132558.log 2023-12-16T11:52:08.391 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113120.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113120.log.gz 2023-12-16T11:52:08.393 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112316.log 2023-12-16T11:52:08.393 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132558.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132558.log.gz 2023-12-16T11:52:08.393 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129549.log 2023-12-16T11:52:08.393 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112316.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112316.log.gz 2023-12-16T11:52:08.393 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98033.log 2023-12-16T11:52:08.394 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129549.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129549.log.gz 2023-12-16T11:52:08.394 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116447.log 2023-12-16T11:52:08.397 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98033.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98033.log.gz 2023-12-16T11:52:08.397 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106344.log 2023-12-16T11:52:08.397 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116447.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.116447.log.gz 2023-12-16T11:52:08.397 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113608.log 2023-12-16T11:52:08.397 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106344.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106344.log.gz 2023-12-16T11:52:08.398 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104998.log 2023-12-16T11:52:08.399 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113608.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.105253.log.gz 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113608.log.gz 2023-12-16T11:52:08.399 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.399 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118394.log 2023-12-16T11:52:08.400 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99562.log 2023-12-16T11:52:08.400 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104998.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104998.log.gz 2023-12-16T11:52:08.400 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114012.log 2023-12-16T11:52:08.401 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99562.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118394.log.gz 2023-12-16T11:52:08.405 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99562.log.gz 2023-12-16T11:52:08.407 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134493.log 2023-12-16T11:52:08.407 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132348.log 2023-12-16T11:52:08.407 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134493.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134493.log.gz 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117392.log 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132348.log: 92.2% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132348.log.gz 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.114012.log.gz 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124685.log 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.117392.log: /var/log/ceph/cluster2-client.admin.136677.log 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117392.log.gz/var/log/ceph/cluster2-client.mirror.124685.log: 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107862.log 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136677.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136677.log.gz 2023-12-16T11:52:08.412 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118561.log 2023-12-16T11:52:08.413 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107862.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107862.log.gz 2023-12-16T11:52:08.413 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126211.log 2023-12-16T11:52:08.414 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118561.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118561.log.gz 2023-12-16T11:52:08.415 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.91564.log 2023-12-16T11:52:08.415 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126211.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126211.log.gz 2023-12-16T11:52:08.415 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99476.log 2023-12-16T11:52:08.428 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.91564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93509.log 2023-12-16T11:52:08.430 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99476.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104741.log.gz 2023-12-16T11:52:08.434 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99476.log.gz 2023-12-16T11:52:08.434 INFO:teuthology.orchestra.run.smithi160.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.1.91564.log.gz 2023-12-16T11:52:08.434 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111055.log 2023-12-16T11:52:08.434 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.93509.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.93509.log.gz 2023-12-16T11:52:08.435 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115592.log 2023-12-16T11:52:08.435 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127518.log 2023-12-16T11:52:08.437 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111055.log: /var/log/ceph/cluster1-client.mirror.115592.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111055.log.gz 2023-12-16T11:52:08.437 INFO:teuthology.orchestra.run.smithi160.stderr: 92.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107799.log 2023-12-16T11:52:08.438 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.115592.log.gz 2023-12-16T11:52:08.438 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127518.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127518.log.gz 2023-12-16T11:52:08.439 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102384.log 2023-12-16T11:52:08.439 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100715.log 2023-12-16T11:52:08.439 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107799.log: /var/log/ceph/cluster2-client.mirror.102384.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.107799.log.gz 2023-12-16T11:52:08.440 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99931.log 2023-12-16T11:52:08.449 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133146.log 2023-12-16T11:52:08.463 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99931.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.91640.log.gz 2023-12-16T11:52:08.463 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.99931.log.gz 2023-12-16T11:52:08.463 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112224.log 2023-12-16T11:52:08.463 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133146.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133146.log.gz 2023-12-16T11:52:08.464 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127497.log 2023-12-16T11:52:08.476 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112224.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112224.log.gz 2023-12-16T11:52:08.476 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119904.log 2023-12-16T11:52:08.476 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.127497.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127497.log.gz 2023-12-16T11:52:08.476 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125416.log 2023-12-16T11:52:08.476 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119904.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119904.log.gz 2023-12-16T11:52:08.476 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109613.log 2023-12-16T11:52:08.476 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125416.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125416.log.gz 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125884.log 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109613.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109613.log.gz 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92640.log 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125884.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125884.log.gz 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124797.log 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92640.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92640.log.gz 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97961.log 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124797.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124797.log.gz 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137559.log 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97961.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97961.log.gz 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118146.log 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137559.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137559.log.gz 2023-12-16T11:52:08.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130878.log 2023-12-16T11:52:08.479 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118146.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118146.log.gz 2023-12-16T11:52:08.479 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98917.log 2023-12-16T11:52:08.479 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130878.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130878.log.gz 2023-12-16T11:52:08.480 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104919.log 2023-12-16T11:52:08.485 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98917.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.98917.log.gz 2023-12-16T11:52:08.485 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132012.log 2023-12-16T11:52:08.485 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104919.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104919.log.gz 2023-12-16T11:52:08.489 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98185.log 2023-12-16T11:52:08.489 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132012.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132012.log.gz 2023-12-16T11:52:08.489 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93565.log 2023-12-16T11:52:08.497 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98185.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102937.log.gz 2023-12-16T11:52:08.500 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125272.log 2023-12-16T11:52:08.501 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.93565.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93565.log.gz 2023-12-16T11:52:08.512 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133062.log 2023-12-16T11:52:08.513 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125272.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125272.log.gz 2023-12-16T11:52:08.515 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102070.log 2023-12-16T11:52:08.515 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133062.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133062.log.gz 2023-12-16T11:52:08.515 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90714.log 2023-12-16T11:52:08.516 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102070.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102070.log.gz 2023-12-16T11:52:08.516 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102579.log 2023-12-16T11:52:08.517 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90714.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90714.log.gz 2023-12-16T11:52:08.517 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119139.log 2023-12-16T11:52:08.518 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102579.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.102579.log.gz 2023-12-16T11:52:08.518 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95865.log 2023-12-16T11:52:08.519 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119139.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119139.log.gz 2023-12-16T11:52:08.520 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104978.log 2023-12-16T11:52:08.521 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95865.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95865.log.gz 2023-12-16T11:52:08.521 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134113.log 2023-12-16T11:52:08.534 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99109.log 2023-12-16T11:52:08.534 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134113.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134113.log.gz 2023-12-16T11:52:08.548 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128408.log 2023-12-16T11:52:08.562 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99109.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99109.log.gz 2023-12-16T11:52:08.566 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124481.log 2023-12-16T11:52:08.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128408.log: 76.1% -- replaced with /var/log/ceph/cluster2-client.mirror.128408.log.gz 2023-12-16T11:52:08.578 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115949.log 2023-12-16T11:52:08.579 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124481.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124481.log.gz 2023-12-16T11:52:08.591 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101636.log.gz 2023-12-16T11:52:08.591 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90627.log 2023-12-16T11:52:08.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115949.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115949.log.gz 2023-12-16T11:52:08.591 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115004.log 2023-12-16T11:52:08.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90627.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90627.log.gz 2023-12-16T11:52:08.592 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94969.log 2023-12-16T11:52:08.592 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115004.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115004.log.gz 2023-12-16T11:52:08.592 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129954.log 2023-12-16T11:52:08.596 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94969.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94969.log.gz 2023-12-16T11:52:08.596 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136383.log 2023-12-16T11:52:08.598 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.129954.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129954.log.gz 2023-12-16T11:52:08.598 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100955.log 2023-12-16T11:52:08.598 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136383.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136383.log.gz 2023-12-16T11:52:08.599 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118539.log 2023-12-16T11:52:08.599 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100955.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100955.log.gz 2023-12-16T11:52:08.602 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127453.log 2023-12-16T11:52:08.602 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118539.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118539.log.gz 2023-12-16T11:52:08.602 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.103771.log 2023-12-16T11:52:08.606 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127453.log: 93.7% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101402.log.gz 2023-12-16T11:52:08.606 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.94576.log.gz 2023-12-16T11:52:08.606 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117619.log 2023-12-16T11:52:08.606 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.3.103771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107099.log 2023-12-16T11:52:08.607 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117619.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117619.log.gz 2023-12-16T11:52:08.607 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119553.log 2023-12-16T11:52:08.607 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107099.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107099.log.gz 2023-12-16T11:52:08.607 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122202.log 2023-12-16T11:52:08.609 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119553.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119553.log.gz 2023-12-16T11:52:08.609 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139155.log 2023-12-16T11:52:08.610 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122202.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122202.log.gz 2023-12-16T11:52:08.610 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107164.log 2023-12-16T11:52:08.610 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139155.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139155.log.gz 2023-12-16T11:52:08.610 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127431.log 2023-12-16T11:52:08.611 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107164.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107164.log.gz 2023-12-16T11:52:08.611 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122157.log 2023-12-16T11:52:08.612 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127431.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127431.log.gz 2023-12-16T11:52:08.612 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128250.log 2023-12-16T11:52:08.613 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122157.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122157.log.gz 2023-12-16T11:52:08.613 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111167.log 2023-12-16T11:52:08.614 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128250.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128250.log.gz 2023-12-16T11:52:08.614 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104936.log 2023-12-16T11:52:08.615 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111167.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111167.log.gz 2023-12-16T11:52:08.615 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103152.log 2023-12-16T11:52:08.616 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104936.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104936.log.gz 2023-12-16T11:52:08.616 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139575.log 2023-12-16T11:52:08.619 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103152.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103152.log.gz 2023-12-16T11:52:08.620 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.100028.log 2023-12-16T11:52:08.620 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139575.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139575.log.gz 2023-12-16T11:52:08.620 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98276.log 2023-12-16T11:52:08.624 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.0.100028.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100715.log.gz 2023-12-16T11:52:08.633 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127747.log 2023-12-16T11:52:08.634 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98276.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98276.log.gz 2023-12-16T11:52:08.635 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137055.log 2023-12-16T11:52:08.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127747.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127747.log.gz 2023-12-16T11:52:08.638 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105253.log 2023-12-16T11:52:08.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137055.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137055.log.gz 2023-12-16T11:52:08.638 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129996.log 2023-12-16T11:52:08.640 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105253.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105253.log.gz 2023-12-16T11:52:08.640 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109519.log 2023-12-16T11:52:08.641 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.129996.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.129996.log.gz 2023-12-16T11:52:08.641 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97392.log 2023-12-16T11:52:08.642 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109519.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109519.log.gz 2023-12-16T11:52:08.642 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122315.log 2023-12-16T11:52:08.644 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97392.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97392.log.gz 2023-12-16T11:52:08.644 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119387.log 2023-12-16T11:52:08.645 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122315.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122315.log.gz 2023-12-16T11:52:08.645 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113666.log 2023-12-16T11:52:08.647 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119387.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119387.log.gz 2023-12-16T11:52:08.647 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.100028.log.gz 2023-12-16T11:52:08.647 INFO:teuthology.orchestra.run.smithi160.stderr: -- /var/log/ceph/cluster2-client.mirror.116914.log 2023-12-16T11:52:08.648 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99539.log 2023-12-16T11:52:08.648 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113666.log: /var/log/ceph/cluster2-client.mirror.116914.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113666.log.gz 2023-12-16T11:52:08.649 INFO:teuthology.orchestra.run.smithi160.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116914.log.gz 2023-12-16T11:52:08.650 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98141.log 2023-12-16T11:52:08.650 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108239.log 2023-12-16T11:52:08.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98141.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99539.log.gz 2023-12-16T11:52:08.657 INFO:teuthology.orchestra.run.smithi160.stderr: 93.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132474.log 2023-12-16T11:52:08.657 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98141.log.gz 2023-12-16T11:52:08.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.125520.log 2023-12-16T11:52:08.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132474.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108239.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132474.log.gz 2023-12-16T11:52:08.657 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.657 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118353.log 2023-12-16T11:52:08.659 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132138.log 2023-12-16T11:52:08.659 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.3.125520.log: /var/log/ceph/cluster2-client.mirror.118353.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118353.log.gz 2023-12-16T11:52:08.659 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115230.log 2023-12-16T11:52:08.659 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132138.log: 0.0% 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.3.103771.log.gz 2023-12-16T11:52:08.659 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.132138.log.gz 2023-12-16T11:52:08.659 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96463.log 2023-12-16T11:52:08.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85750.log 2023-12-16T11:52:08.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96463.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115230.log.gz 2023-12-16T11:52:08.661 INFO:teuthology.orchestra.run.smithi160.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96463.log.gz 2023-12-16T11:52:08.661 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118685.log 2023-12-16T11:52:08.661 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125863.log 2023-12-16T11:52:08.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.85750.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85750.log.gz 2023-12-16T11:52:08.662 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.123463.log 2023-12-16T11:52:08.662 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118685.log.gz 2023-12-16T11:52:08.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125863.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.125520.log.gz 2023-12-16T11:52:08.663 INFO:teuthology.orchestra.run.smithi160.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125863.log.gz 2023-12-16T11:52:08.664 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137517.log 2023-12-16T11:52:08.664 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103877.log 2023-12-16T11:52:08.664 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.1.123463.log: /var/log/ceph/cluster2-client.admin.137517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113300.log 2023-12-16T11:52:08.664 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137517.log.gz 2023-12-16T11:52:08.665 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103877.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103877.log.gz 2023-12-16T11:52:08.666 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102384.log.gz 2023-12-16T11:52:08.682 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97867.log 2023-12-16T11:52:08.682 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104427.log 2023-12-16T11:52:08.682 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97867.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113300.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124052.log 2023-12-16T11:52:08.682 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.682 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128291.log 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104427.log: /var/log/ceph/cluster2-client.mirror.124052.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97867.log.gz 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104427.log.gz 90.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103132.log 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.124052.log.gz 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120560.log 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109018.log 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120560.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103132.log.gz 2023-12-16T11:52:08.683 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120560.log.gz 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128291.log.gz 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102858.log 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99711.log 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102858.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109018.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118912.log 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127165.log 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99711.log: /var/log/ceph/cluster2-client.mirror.118912.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118912.log.gz 88.2% 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.99711.log.gz 2023-12-16T11:52:08.684 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.123463.log.gz 2023-12-16T11:52:08.685 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139113.log 2023-12-16T11:52:08.685 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104035.log 2023-12-16T11:52:08.685 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127165.log: /var/log/ceph/cluster2-client.admin.139113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102757.log 2023-12-16T11:52:08.685 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139113.log.gz 2023-12-16T11:52:08.685 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127165.log.gz 2023-12-16T11:52:08.685 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119119.log 2023-12-16T11:52:08.685 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100838.log 2023-12-16T11:52:08.686 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102757.log.gz/var/log/ceph/cluster2-client.mirror.119119.log: 2023-12-16T11:52:08.686 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119346.log 2023-12-16T11:52:08.686 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100838.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119119.log.gz 2023-12-16T11:52:08.686 INFO:teuthology.orchestra.run.smithi160.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100838.log.gz 2023-12-16T11:52:08.686 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138231.log 2023-12-16T11:52:08.686 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91386.log 2023-12-16T11:52:08.686 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138231.log: /var/log/ceph/cluster2-client.mirror.119346.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138231.log.gz 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129028.log 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119346.log.gz 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.91386.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91386.log.gz 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134787.log 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137265.log 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129028.log: /var/log/ceph/cluster2-client.admin.134787.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134787.log.gz 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129028.log.gz 2023-12-16T11:52:08.687 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128929.log 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118581.log 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137265.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137265.log.gz 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.91510.log 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128929.log.gz 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136257.log 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.0.91510.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118581.log.gz 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136467.log 2023-12-16T11:52:08.688 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136257.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136257.log.gz 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134155.log 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136467.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136467.log.gz 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93873.log 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134155.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134155.log.gz 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119532.log 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.91510.log.gz 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115720.log 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119532.log: 74.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93873.log.gz 2023-12-16T11:52:08.714 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110231.log 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119532.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108030.log 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110231.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115720.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117185.log 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110231.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132810.log 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117185.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108030.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92713.log 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.117185.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/cluster2-client.admin.132810.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.94002.log 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132810.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140289.log 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92713.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92713.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.1.94002.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104978.log.gz 2023-12-16T11:52:08.715 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140289.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140289.log.gz 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr: -- /var/log/ceph/cluster2-client.mirror.125105.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114458.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109139.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr: 85.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125105.log.gz 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108396.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109139.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109139.log.gz 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114892.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr: 93.1%/var/log/ceph/cluster1-client.mirror.108396.log: -- replaced with /var/log/ceph/cluster2-client.mirror.1.94002.log.gz 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137307.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114892.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114458.log.gz 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122540.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108396.log.gz/var/log/ceph/cluster2-client.admin.137307.log: 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137307.log.gz 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113142.log 2023-12-16T11:52:08.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122540.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114892.log.gz 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106109.log 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122540.log.gz 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/cluster2-client.mirror.113142.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103939.log 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.101885.log 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113142.log.gz 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106109.log.gz 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109163.log 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr: 79.5%/var/log/ceph/cluster2-client.mirror.0.101885.log: -- replaced with /var/log/ceph/cluster2-client.mirror.103939.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91218.log 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110860.log 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.91218.log: /var/log/ceph/cluster1-client.mirror.109163.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91218.log.gz 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109163.log.gz 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131760.log 2023-12-16T11:52:08.717 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110860.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110860.log.gz 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106935.log 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131760.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134955.log 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.131760.log.gz 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94948.log 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134955.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134955.log.gz 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94630.log 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94948.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94948.log.gz 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106935.log.gz 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110069.log 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94630.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94630.log.gz 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127604.log 2023-12-16T11:52:08.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99991.log 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127604.log: 86.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127604.log.gz 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91428.log 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99991.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110069.log.gz 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99991.log.gz 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105365.log 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.91428.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91428.log.gz 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98874.log 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104662.log 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98874.log: /var/log/ceph/cluster2-client.mirror.105365.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105365.log.gz 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103576.log 2023-12-16T11:52:08.719 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104662.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.98874.log.gz 2023-12-16T11:52:08.720 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119842.log 2023-12-16T11:52:08.720 INFO:teuthology.orchestra.run.smithi087.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-12-16T11:52:08.725 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113252.log 2023-12-16T11:52:08.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119842.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119842.log.gz 2023-12-16T11:52:08.736 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120582.log 2023-12-16T11:52:08.736 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113252.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113252.log.gz 2023-12-16T11:52:08.748 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128868.log 2023-12-16T11:52:08.749 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120582.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120582.log.gz 2023-12-16T11:52:08.762 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113497.log 2023-12-16T11:52:08.762 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128868.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128868.log.gz 2023-12-16T11:52:08.773 INFO:teuthology.orchestra.run.smithi160.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.104035.log.gz 2023-12-16T11:52:08.773 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130122.log 2023-12-16T11:52:08.773 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113497.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113497.log.gz 2023-12-16T11:52:08.776 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138399.log 2023-12-16T11:52:08.776 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130122.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130122.log.gz 2023-12-16T11:52:08.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104407.log 2023-12-16T11:52:08.777 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138399.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138399.log.gz 2023-12-16T11:52:08.777 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130374.log 2023-12-16T11:52:08.789 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119243.log 2023-12-16T11:52:08.789 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130374.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130374.log.gz 2023-12-16T11:52:08.798 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124685.log.gz 2023-12-16T11:52:08.801 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.101885.log.gz 2023-12-16T11:52:08.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127624.log 2023-12-16T11:52:08.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124663.log 2023-12-16T11:52:08.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119243.log: /var/log/ceph/cluster2-client.mirror.127624.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.119243.log.gz 2023-12-16T11:52:08.805 INFO:teuthology.orchestra.run.smithi160.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127624.log.gz 2023-12-16T11:52:08.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90734.log 2023-12-16T11:52:08.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129643.log 2023-12-16T11:52:08.806 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124663.log: /var/log/ceph/cluster2-client.admin.90734.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90734.log.gz 2023-12-16T11:52:08.807 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124663.log.gz 2023-12-16T11:52:08.807 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98489.log 2023-12-16T11:52:08.807 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91050.log 2023-12-16T11:52:08.809 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.129643.log: /var/log/ceph/cluster1-client.mirror.98489.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129643.log.gz 2023-12-16T11:52:08.809 INFO:teuthology.orchestra.run.smithi160.stderr:gzip 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98489.log.gz 2023-12-16T11:52:08.809 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106181.log 2023-12-16T11:52:08.809 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97550.log 2023-12-16T11:52:08.809 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.91050.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91050.log.gz 2023-12-16T11:52:08.810 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102779.log 2023-12-16T11:52:08.810 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97550.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106181.log.gz 2023-12-16T11:52:08.810 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122292.log 2023-12-16T11:52:08.816 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102779.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97550.log.gz 2023-12-16T11:52:08.820 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102033.log 2023-12-16T11:52:08.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122292.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122292.log.gz 2023-12-16T11:52:08.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113586.log 2023-12-16T11:52:08.835 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122405.log 2023-12-16T11:52:08.837 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113586.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113586.log.gz 2023-12-16T11:52:08.842 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102500.log 2023-12-16T11:52:08.855 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122405.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106364.log 2023-12-16T11:52:08.855 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122405.log.gz 2023-12-16T11:52:08.856 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102500.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102500.log.gz 2023-12-16T11:52:08.869 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112834.log 2023-12-16T11:52:08.870 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106364.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106364.log.gz 2023-12-16T11:52:08.870 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127453.log.gz 2023-12-16T11:52:08.873 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122247.log 2023-12-16T11:52:08.873 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112834.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112834.log.gz 2023-12-16T11:52:08.873 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113363.log 2023-12-16T11:52:08.873 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122247.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122247.log.gz 2023-12-16T11:52:08.873 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128349.log 2023-12-16T11:52:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113363.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113363.log.gz 2023-12-16T11:52:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139491.log 2023-12-16T11:52:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128349.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128349.log.gz 2023-12-16T11:52:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119718.log 2023-12-16T11:52:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139491.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139491.log.gz 2023-12-16T11:52:08.876 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116845.log 2023-12-16T11:52:08.878 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119718.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119718.log.gz 2023-12-16T11:52:08.879 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97745.log 2023-12-16T11:52:08.879 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116845.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116845.log.gz 2023-12-16T11:52:08.879 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133524.log 2023-12-16T11:52:08.879 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97745.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97745.log.gz 2023-12-16T11:52:08.881 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98513.log 2023-12-16T11:52:08.881 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133524.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133524.log.gz 2023-12-16T11:52:08.881 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113875.log 2023-12-16T11:52:08.881 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98513.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98513.log.gz 2023-12-16T11:52:08.882 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127474.log 2023-12-16T11:52:08.884 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113875.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113875.log.gz 2023-12-16T11:52:08.884 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116797.log 2023-12-16T11:52:08.884 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.127474.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127474.log.gz 2023-12-16T11:52:08.884 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131466.log 2023-12-16T11:52:08.886 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116797.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116797.log.gz 2023-12-16T11:52:08.886 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-12-16T11:52:08.887 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131466.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131466.log.gz 2023-12-16T11:52:08.887 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126252.log 2023-12-16T11:52:08.894 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-osd.1.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98185.log.gz 2023-12-16T11:52:08.896 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102858.log.gz 2023-12-16T11:52:08.898 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104662.log.gz 2023-12-16T11:52:08.904 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113825.log 2023-12-16T11:52:08.930 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126252.log: gzip 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126252.log.gz 2023-12-16T11:52:08.930 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128312.log 2023-12-16T11:52:08.930 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139827.log 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128312.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113825.log.gz 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128312.log.gz 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr: -- /var/log/ceph/cluster2-client.admin.133566.log 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139827.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111077.log 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.139827.log.gz 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133566.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106778.log 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133566.log.gz 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.111077.log: --verbose -- /var/log/ceph/cluster2-client.mirror.120276.log 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140247.log 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111077.log.gz 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120276.log: gzip 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106778.log.gz 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102915.log 2023-12-16T11:52:08.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140247.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114365.log 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.140247.log.gz 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120276.log.gz 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102915.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102915.log.gz 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115028.log 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.98247.log 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115028.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114365.log.gz 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90196.log 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110613.log 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.115028.log.gz 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.2.98247.log: /var/log/ceph/cluster2-client.admin.90196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120920.log 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110613.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90196.log.gz 2023-12-16T11:52:08.932 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129570.log 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120920.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120920.log.gz 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110613.log.gz 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90252.log 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133104.log 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90252.log: /var/log/ceph/cluster2-client.mirror.129570.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90252.log.gz 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129570.log.gz 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118871.log 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133104.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133104.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121100.log 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125740.log 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121100.log: /var/log/ceph/cluster2-client.mirror.118871.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121100.log.gz 2023-12-16T11:52:08.933 INFO:teuthology.orchestra.run.smithi160.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118871.log.gz 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121032.log 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134029.log 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125740.log.gz/var/log/ceph/cluster1-client.mirror.121032.log: 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121032.log.gz 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128724.log 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134029.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134029.log.gz 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130836.log 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96441.log 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130836.log: /var/log/ceph/cluster2-client.mirror.128724.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130836.log.gz 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128724.log.gz 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107027.log 2023-12-16T11:52:08.934 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131424.log 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107027.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107027.log.gz 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112899.log 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131424.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131424.log.gz 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98078.log 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112899.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112899.log.gz 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135543.log 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98078.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96441.log.gz 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99132.log 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135543.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135543.log.gz 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97985.log 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99132.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98078.log.gz 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138357.log 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97985.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97985.log.gz 2023-12-16T11:52:08.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118477.log 2023-12-16T11:52:08.936 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138357.log.gz 2023-12-16T11:52:08.936 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121190.log 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99132.log.gz 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130920.log 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121190.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118477.log.gz 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121190.log.gz 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105695.log 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103475.log 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105695.log: /var/log/ceph/cluster2-client.admin.130920.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130920.log.gz 2023-12-16T11:52:08.978 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132390.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105695.log.gz 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113276.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132390.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103475.log.gz 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132390.log.gz 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109972.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111726.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113276.log.gz 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117309.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111726.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109972.log.gz 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102974.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117309.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117309.log.gz 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118705.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102974.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111726.log.gz 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120235.log 2023-12-16T11:52:08.979 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118705.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102974.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107505.log 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120235.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120235.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118705.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109637.log 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94852.log 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109637.log: /var/log/ceph/cluster1-client.mirror.107505.log: 87.3% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107505.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109637.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131004.log 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94852.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.98247.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113690.log 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94852.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131004.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131004.log.gz 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124753.log 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106460.log 2023-12-16T11:52:08.980 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.124753.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113690.log.gz 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130458.log 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124753.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108797.log 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106460.log: /var/log/ceph/cluster2-client.admin.130458.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130458.log.gz 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106460.log.gz 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108994.log 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127000.log 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108994.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108797.log.gz 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131340.log 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108994.log.gz 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111239.log 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.127000.log: /var/log/ceph/cluster2-client.admin.131340.log: 0.0% 86.9% -- replaced with /var/log/ceph/cluster2-client.admin.131340.log.gz 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.127000.log.gz 2023-12-16T11:52:08.981 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131130.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118643.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131130.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111239.log.gz 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131130.log.gz 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100219.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/cluster2-client.mirror.118643.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104543.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94469.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104543.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104543.log.gz 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118643.log.gz 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120028.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr: 90.2%/var/log/ceph/cluster2-client.mirror.94469.log: -- replaced with /var/log/ceph/cluster2-client.mirror.100219.log.gz 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128329.log 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-mon.a.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120028.log.gz 2023-12-16T11:52:08.982 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117578.log 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128329.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128329.log.gz 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106436.log 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117578.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117578.log.gz 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133230.log 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106436.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106436.log.gz 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100673.log 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr: 93.2%/var/log/ceph/cluster2-client.admin.133230.log: -- replaced with /var/log/ceph/cluster2-client.mirror.94469.log.gz 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133230.log.gz 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110347.log 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131802.log 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110347.log: 88.7% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110347.log.gz 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100673.log.gz 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111191.log 2023-12-16T11:52:08.983 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125520.log 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131802.log.gz 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99625.log 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111191.log.gz/var/log/ceph/cluster1-client.mirror.125520.log: 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr: 82.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125520.log.gz 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124944.log 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105844.log 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124944.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124944.log.gz 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100875.log 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105844.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105844.log.gz 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114241.log 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100875.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100875.log.gz 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110548.log 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114241.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99625.log.gz 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114241.log.gz 2023-12-16T11:52:08.984 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114823.log 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96738.log 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114823.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110548.log.gz 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106798.log 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114823.log.gz 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.96738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138105.log 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106798.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.96738.log.gz 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115163.log 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106798.log.gz 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138105.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138105.log.gz 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115389.log 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128680.log 2023-12-16T11:52:08.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115389.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115163.log.gz 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121392.log 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115389.log.gz 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.128680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107641.log 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128680.log.gz 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121392.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124903.log 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.121392.log.gz 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135585.log 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124903.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107641.log.gz 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124903.log.gz 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111501.log 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109059.log 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135585.log.gz 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138483.log 2023-12-16T11:52:08.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109059.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111501.log.gz 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99811.log 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109059.log.gz/var/log/ceph/cluster2-client.admin.138483.log: 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138483.log.gz 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101222.log 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120470.log 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101222.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99811.log.gz 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101222.log.gz 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134367.log 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115326.log 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9%/var/log/ceph/cluster2-client.admin.134367.log: -- replaced with /var/log/ceph/cluster1-client.mirror.120470.log.gz 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134367.log.gz 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104565.log 2023-12-16T11:52:08.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96485.log 2023-12-16T11:52:08.988 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104565.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115326.log.gz 2023-12-16T11:52:08.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120785.log 2023-12-16T11:52:08.994 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105782.log 2023-12-16T11:52:08.995 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120785.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120785.log.gz 2023-12-16T11:52:09.008 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115652.log 2023-12-16T11:52:09.008 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105782.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105782.log.gz 2023-12-16T11:52:09.013 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104407.log.gz 2023-12-16T11:52:09.021 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103310.log 2023-12-16T11:52:09.022 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115652.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115652.log.gz 2023-12-16T11:52:09.023 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104348.log 2023-12-16T11:52:09.024 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103310.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103310.log.gz 2023-12-16T11:52:09.024 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99324.log 2023-12-16T11:52:09.025 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104348.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104348.log.gz 2023-12-16T11:52:09.025 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108576.log 2023-12-16T11:52:09.031 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99324.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99324.log.gz 2023-12-16T11:52:09.031 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109661.log 2023-12-16T11:52:09.032 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108576.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108576.log.gz 2023-12-16T11:52:09.032 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137391.log 2023-12-16T11:52:09.033 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109661.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109661.log.gz 2023-12-16T11:52:09.034 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.127342.log 2023-12-16T11:52:09.034 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137391.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137391.log.gz 2023-12-16T11:52:09.035 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105985.log 2023-12-16T11:52:09.038 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.3.127342.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.127342.log.gz 2023-12-16T11:52:09.038 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103860.log 2023-12-16T11:52:09.040 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105985.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105985.log.gz 2023-12-16T11:52:09.040 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103534.log 2023-12-16T11:52:09.041 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103860.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103860.log.gz 2023-12-16T11:52:09.041 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112786.log 2023-12-16T11:52:09.043 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103534.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103534.log.gz 2023-12-16T11:52:09.043 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-12-16T11:52:09.044 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112786.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112786.log.gz 2023-12-16T11:52:09.044 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126564.log 2023-12-16T11:52:09.050 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2.log: 88.2% -- replaced with /var/log/ceph/cluster2.log.gz 2023-12-16T11:52:09.050 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100051.log 2023-12-16T11:52:09.051 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126564.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126564.log.gz 2023-12-16T11:52:09.051 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95887.log 2023-12-16T11:52:09.052 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100051.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100051.log.gz 2023-12-16T11:52:09.052 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121505.log 2023-12-16T11:52:09.066 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122450.log 2023-12-16T11:52:09.066 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121505.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121505.log.gz 2023-12-16T11:52:09.080 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118809.log 2023-12-16T11:52:09.080 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122450.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122450.log.gz 2023-12-16T11:52:09.081 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133734.log 2023-12-16T11:52:09.088 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118809.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119945.log 2023-12-16T11:52:09.088 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118809.log.gz 2023-12-16T11:52:09.101 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133734.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139617.log 2023-12-16T11:52:09.101 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.133734.log.gz 2023-12-16T11:52:09.102 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119945.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119945.log.gz 2023-12-16T11:52:09.108 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107461.log 2023-12-16T11:52:09.108 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139617.log.gz 2023-12-16T11:52:09.117 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119863.log 2023-12-16T11:52:09.117 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107461.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107461.log.gz 2023-12-16T11:52:09.134 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.125105.log 2023-12-16T11:52:09.134 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119863.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119863.log.gz 2023-12-16T11:52:09.143 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137727.log 2023-12-16T11:52:09.143 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.3.125105.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104565.log.gz 2023-12-16T11:52:09.146 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134745.log 2023-12-16T11:52:09.146 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137727.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137727.log.gz 2023-12-16T11:52:09.146 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111035.log 2023-12-16T11:52:09.146 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134745.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134745.log.gz 2023-12-16T11:52:09.146 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111995.log 2023-12-16T11:52:09.148 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111035.log: 93.0% 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111035.log.gz 2023-12-16T11:52:09.148 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112858.log 2023-12-16T11:52:09.150 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111995.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111995.log.gz 2023-12-16T11:52:09.150 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.3.125105.log.gz 2023-12-16T11:52:09.150 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114124.log 2023-12-16T11:52:09.151 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112858.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112858.log.gz 2023-12-16T11:52:09.152 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91790.log 2023-12-16T11:52:09.152 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114124.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114124.log.gz 2023-12-16T11:52:09.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104798.log 2023-12-16T11:52:09.156 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.91790.log: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.91790.log.gz 2023-12-16T11:52:09.156 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102779.log.gz 2023-12-16T11:52:09.156 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98057.log 2023-12-16T11:52:09.156 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90628.log 2023-12-16T11:52:09.156 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98057.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104798.log.gz 2023-12-16T11:52:09.156 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127809.log 2023-12-16T11:52:09.156 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98057.log.gz 2023-12-16T11:52:09.158 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90628.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90628.log.gz 2023-12-16T11:52:09.158 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139953.log 2023-12-16T11:52:09.158 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134325.log 2023-12-16T11:52:09.158 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127809.log: /var/log/ceph/cluster2-client.admin.139953.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139953.log.gz 2023-12-16T11:52:09.158 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127809.log.gz 2023-12-16T11:52:09.158 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107369.log 2023-12-16T11:52:09.159 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107144.log 2023-12-16T11:52:09.159 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134325.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134325.log.gz 2023-12-16T11:52:09.159 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106065.log 2023-12-16T11:52:09.162 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107144.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107369.log.gz 2023-12-16T11:52:09.162 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139281.log 2023-12-16T11:52:09.162 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106065.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107144.log.gz 2023-12-16T11:52:09.162 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94600.log 2023-12-16T11:52:09.162 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139281.log: 88.9% 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106065.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.139281.log.gz 2023-12-16T11:52:09.162 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.162 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90378.log 2023-12-16T11:52:09.163 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92093.log 2023-12-16T11:52:09.163 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.94600.log: /var/log/ceph/cluster2-client.admin.90378.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90378.log.gz 2023-12-16T11:52:09.163 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.94600.log.gz 2023-12-16T11:52:09.163 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126313.log 2023-12-16T11:52:09.164 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96946.log 2023-12-16T11:52:09.165 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.92093.log: 66.4% -- replaced with /var/log/ceph/cluster1-client.admin.92093.log.gz 2023-12-16T11:52:09.165 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97680.log 2023-12-16T11:52:09.165 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126313.log.gz 2023-12-16T11:52:09.165 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96946.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96946.log.gz 2023-12-16T11:52:09.166 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132600.log 2023-12-16T11:52:09.166 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135501.log 2023-12-16T11:52:09.166 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97680.log: /var/log/ceph/cluster2-client.admin.132600.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132600.log.gz 2023-12-16T11:52:09.170 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95076.log 2023-12-16T11:52:09.170 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135501.log.gz 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105407.log 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97680.log.gz 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114558.log 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105407.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105407.log.gz 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93653.log 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114558.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114558.log.gz 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119739.log 2023-12-16T11:52:09.174 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93653.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93653.log.gz 2023-12-16T11:52:09.176 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124605.log 2023-12-16T11:52:09.176 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119739.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119739.log.gz 2023-12-16T11:52:09.176 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121235.log 2023-12-16T11:52:09.178 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124605.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124605.log.gz 2023-12-16T11:52:09.178 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113473.log 2023-12-16T11:52:09.178 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121235.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121235.log.gz 2023-12-16T11:52:09.178 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114478.log 2023-12-16T11:52:09.180 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113473.log: 87.4% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103576.log.gz 2023-12-16T11:52:09.180 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113473.log.gz 2023-12-16T11:52:09.180 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122135.log 2023-12-16T11:52:09.180 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/cluster1-client.mirror.114478.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103290.log 2023-12-16T11:52:09.182 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122135.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122135.log.gz 2023-12-16T11:52:09.182 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99891.log 2023-12-16T11:52:09.182 INFO:teuthology.orchestra.run.smithi160.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114478.log.gz/var/log/ceph/cluster2-client.mirror.103290.log: 2023-12-16T11:52:09.182 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96898.log 2023-12-16T11:52:09.182 INFO:teuthology.orchestra.run.smithi160.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103290.log.gz 2023-12-16T11:52:09.182 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99891.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99891.log.gz 2023-12-16T11:52:09.185 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106677.log 2023-12-16T11:52:09.185 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125294.log 2023-12-16T11:52:09.185 INFO:teuthology.orchestra.run.smithi160.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.96898.log.gz/var/log/ceph/cluster1-client.mirror.106677.log: 2023-12-16T11:52:09.185 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96285.log 2023-12-16T11:52:09.185 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125294.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106677.log.gz 2023-12-16T11:52:09.185 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129300.log 2023-12-16T11:52:09.185 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96285.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.96285.log.gz 2023-12-16T11:52:09.186 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116285.log 2023-12-16T11:52:09.186 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.129300.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.129300.log.gz 2023-12-16T11:52:09.186 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114775.log 2023-12-16T11:52:09.187 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116285.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116285.log.gz 2023-12-16T11:52:09.187 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124358.log 2023-12-16T11:52:09.188 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114775.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114775.log.gz 2023-12-16T11:52:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124924.log 2023-12-16T11:52:09.189 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124358.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124358.log.gz 2023-12-16T11:52:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138987.log 2023-12-16T11:52:09.190 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124924.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124924.log.gz 2023-12-16T11:52:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107703.log 2023-12-16T11:52:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138987.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138987.log.gz 2023-12-16T11:52:09.191 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102895.log 2023-12-16T11:52:09.192 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107703.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107703.log.gz 2023-12-16T11:52:09.192 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93794.log 2023-12-16T11:52:09.193 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102895.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102895.log.gz 2023-12-16T11:52:09.194 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138777.log 2023-12-16T11:52:09.199 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93794.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93794.log.gz 2023-12-16T11:52:09.200 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131046.log 2023-12-16T11:52:09.200 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138777.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138777.log.gz 2023-12-16T11:52:09.200 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104272.log 2023-12-16T11:52:09.201 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131046.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131046.log.gz 2023-12-16T11:52:09.201 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126832.log 2023-12-16T11:52:09.214 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126043.log 2023-12-16T11:52:09.215 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126832.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126832.log.gz 2023-12-16T11:52:09.228 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126977.log 2023-12-16T11:52:09.228 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126043.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126043.log.gz 2023-12-16T11:52:09.240 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112674.log 2023-12-16T11:52:09.241 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126977.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126977.log.gz 2023-12-16T11:52:09.243 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102033.log.gz 2023-12-16T11:52:09.244 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96357.log 2023-12-16T11:52:09.246 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112674.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112674.log.gz 2023-12-16T11:52:09.247 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102421.log 2023-12-16T11:52:09.247 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96357.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.96357.log.gz 2023-12-16T11:52:09.248 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92229.log 2023-12-16T11:52:09.249 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102421.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102421.log.gz 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117681.log 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.92229.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92229.log.gz 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94400.log 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117681.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117681.log.gz 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113007.log 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94400.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94400.log.gz 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126064.log 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113007.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113007.log.gz 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99390.log 2023-12-16T11:52:09.256 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126064.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126064.log.gz 2023-12-16T11:52:09.257 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107899.log 2023-12-16T11:52:09.262 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99390.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99390.log.gz 2023-12-16T11:52:09.262 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110593.log 2023-12-16T11:52:09.264 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107899.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107899.log.gz 2023-12-16T11:52:09.264 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.124052.log 2023-12-16T11:52:09.266 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110593.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110593.log.gz 2023-12-16T11:52:09.266 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117454.log 2023-12-16T11:52:09.280 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.2.124052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103956.log 2023-12-16T11:52:09.280 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117454.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117454.log.gz 2023-12-16T11:52:09.288 INFO:teuthology.orchestra.run.smithi160.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120359.log 2023-12-16T11:52:09.290 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103956.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103956.log.gz 2023-12-16T11:52:09.290 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.124052.log.gz 2023-12-16T11:52:09.290 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113967.log 2023-12-16T11:52:09.291 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120359.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120359.log.gz 2023-12-16T11:52:09.291 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105344.log 2023-12-16T11:52:09.292 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113967.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113967.log.gz 2023-12-16T11:52:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97180.log 2023-12-16T11:52:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105344.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105344.log.gz 2023-12-16T11:52:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106388.log 2023-12-16T11:52:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.97180.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.97180.log.gz 2023-12-16T11:52:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106501.log 2023-12-16T11:52:09.296 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106388.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106388.log.gz 2023-12-16T11:52:09.296 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116222.log 2023-12-16T11:52:09.299 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106501.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106501.log.gz 2023-12-16T11:52:09.299 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95663.log 2023-12-16T11:52:09.301 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116222.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.116222.log.gz 2023-12-16T11:52:09.301 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115831.log 2023-12-16T11:52:09.301 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95663.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95663.log.gz 2023-12-16T11:52:09.302 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120830.log 2023-12-16T11:52:09.303 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115831.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115831.log.gz 2023-12-16T11:52:09.303 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121797.log 2023-12-16T11:52:09.307 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120830.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120830.log.gz 2023-12-16T11:52:09.307 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138063.log 2023-12-16T11:52:09.307 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121797.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121797.log.gz 2023-12-16T11:52:09.307 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108639.log 2023-12-16T11:52:09.307 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138063.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138063.log.gz 2023-12-16T11:52:09.307 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110797.log 2023-12-16T11:52:09.307 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108639.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108639.log.gz 2023-12-16T11:52:09.309 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93701.log 2023-12-16T11:52:09.309 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110797.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110797.log.gz 2023-12-16T11:52:09.309 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109360.log 2023-12-16T11:52:09.311 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.93701.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.93701.log.gz 2023-12-16T11:52:09.311 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103369.log 2023-12-16T11:52:09.311 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109360.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109360.log.gz 2023-12-16T11:52:09.312 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95054.log 2023-12-16T11:52:09.313 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103369.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103369.log.gz 2023-12-16T11:52:09.313 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137853.log 2023-12-16T11:52:09.314 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95054.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.95054.log.gz 2023-12-16T11:52:09.315 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107007.log 2023-12-16T11:52:09.315 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137853.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137853.log.gz 2023-12-16T11:52:09.315 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108189.log 2023-12-16T11:52:09.317 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107007.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.107007.log.gz 2023-12-16T11:52:09.317 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119780.log 2023-12-16T11:52:09.318 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108189.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108189.log.gz 2023-12-16T11:52:09.318 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96398.log 2023-12-16T11:52:09.320 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119780.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119780.log.gz 2023-12-16T11:52:09.320 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122517.log 2023-12-16T11:52:09.326 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96398.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96398.log.gz 2023-12-16T11:52:09.326 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100478.log 2023-12-16T11:52:09.327 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122517.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122517.log.gz 2023-12-16T11:52:09.327 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113762.log 2023-12-16T11:52:09.328 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100478.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100478.log.gz 2023-12-16T11:52:09.328 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134661.log 2023-12-16T11:52:09.329 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113762.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113762.log.gz 2023-12-16T11:52:09.330 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128806.log 2023-12-16T11:52:09.330 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134661.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134661.log.gz 2023-12-16T11:52:09.331 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101556.log 2023-12-16T11:52:09.332 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128806.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128806.log.gz 2023-12-16T11:52:09.332 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131928.log 2023-12-16T11:52:09.350 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99519.log 2023-12-16T11:52:09.350 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131928.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131928.log.gz 2023-12-16T11:52:09.364 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135081.log 2023-12-16T11:52:09.369 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99519.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99519.log.gz 2023-12-16T11:52:09.383 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98163.log 2023-12-16T11:52:09.383 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135081.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135081.log.gz 2023-12-16T11:52:09.390 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127247.log 2023-12-16T11:52:09.390 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98163.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.98163.log.gz 2023-12-16T11:52:09.399 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136299.log 2023-12-16T11:52:09.399 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127247.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127247.log.gz 2023-12-16T11:52:09.405 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95076.log.gz 2023-12-16T11:52:09.405 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99433.log 2023-12-16T11:52:09.406 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136299.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136299.log.gz 2023-12-16T11:52:09.406 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96237.log 2023-12-16T11:52:09.411 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99433.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99433.log.gz 2023-12-16T11:52:09.412 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105868.log 2023-12-16T11:52:09.413 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96237.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96237.log.gz 2023-12-16T11:52:09.414 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118035.log 2023-12-16T11:52:09.414 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105868.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105868.log.gz 2023-12-16T11:52:09.414 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115809.log 2023-12-16T11:52:09.415 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118035.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118035.log.gz 2023-12-16T11:52:09.416 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114500.log 2023-12-16T11:52:09.417 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115809.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115809.log.gz 2023-12-16T11:52:09.417 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112561.log 2023-12-16T11:52:09.418 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114500.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114500.log.gz 2023-12-16T11:52:09.419 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103211.log 2023-12-16T11:52:09.419 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112561.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112561.log.gz 2023-12-16T11:52:09.419 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100348.log 2023-12-16T11:52:09.421 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103211.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103211.log.gz 2023-12-16T11:52:09.421 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121595.log 2023-12-16T11:52:09.435 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102297.log 2023-12-16T11:52:09.435 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121595.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121595.log.gz 2023-12-16T11:52:09.447 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95887.log.gz 2023-12-16T11:52:09.448 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119883.log 2023-12-16T11:52:09.449 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.102297.log: /var/log/ceph/cluster2-client.admin.134451.log 2023-12-16T11:52:09.451 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119883.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119883.log.gz 2023-12-16T11:52:09.451 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127979.log 2023-12-16T11:52:09.452 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134451.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134451.log.gz 2023-12-16T11:52:09.452 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119284.log 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127979.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127979.log.gz 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117226.log 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119284.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119284.log.gz 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119801.log 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117226.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117226.log.gz 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94236.log 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119801.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119801.log.gz 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140163.log 2023-12-16T11:52:09.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94236.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94236.log.gz 2023-12-16T11:52:09.461 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120007.log 2023-12-16T11:52:09.462 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140163.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140163.log.gz 2023-12-16T11:52:09.462 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91726.log 2023-12-16T11:52:09.463 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120007.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120007.log.gz 2023-12-16T11:52:09.463 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114283.log 2023-12-16T11:52:09.464 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.91726.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.91726.log.gz 2023-12-16T11:52:09.464 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112742.log 2023-12-16T11:52:09.465 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114283.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114283.log.gz 2023-12-16T11:52:09.466 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101089.log 2023-12-16T11:52:09.466 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112742.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112742.log.gz 2023-12-16T11:52:09.467 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113079.log 2023-12-16T11:52:09.468 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101089.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101089.log.gz 2023-12-16T11:52:09.468 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134071.log 2023-12-16T11:52:09.469 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113079.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113079.log.gz 2023-12-16T11:52:09.471 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112696.log 2023-12-16T11:52:09.471 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134071.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134071.log.gz 2023-12-16T11:52:09.471 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127305.log 2023-12-16T11:52:09.472 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112696.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112696.log.gz 2023-12-16T11:52:09.472 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140205.log 2023-12-16T11:52:09.474 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127305.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127305.log.gz 2023-12-16T11:52:09.474 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111657.log 2023-12-16T11:52:09.475 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140205.log.gz 2023-12-16T11:52:09.475 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106605.log 2023-12-16T11:52:09.476 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111657.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111657.log.gz 2023-12-16T11:52:09.476 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97124.log 2023-12-16T11:52:09.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106605.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106605.log.gz 2023-12-16T11:52:09.478 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108773.log 2023-12-16T11:52:09.479 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97124.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97124.log.gz 2023-12-16T11:52:09.479 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103655.log 2023-12-16T11:52:09.480 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108773.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108773.log.gz 2023-12-16T11:52:09.480 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126687.log 2023-12-16T11:52:09.494 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97436.log 2023-12-16T11:52:09.494 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126687.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126687.log.gz 2023-12-16T11:52:09.501 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96485.log.gz 2023-12-16T11:52:09.507 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111436.log 2023-12-16T11:52:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97436.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97436.log.gz 2023-12-16T11:52:09.510 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103633.log 2023-12-16T11:52:09.510 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111436.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111436.log.gz 2023-12-16T11:52:09.519 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121370.log 2023-12-16T11:52:09.520 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103633.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103633.log.gz 2023-12-16T11:52:09.535 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95100.log 2023-12-16T11:52:09.536 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121370.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121370.log.gz 2023-12-16T11:52:09.543 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93964.log 2023-12-16T11:52:09.556 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.95100.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.admin.95100.log.gz 2023-12-16T11:52:09.564 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115744.log 2023-12-16T11:52:09.585 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95687.log 2023-12-16T11:52:09.585 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115744.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115744.log.gz 2023-12-16T11:52:09.595 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125294.log.gz 2023-12-16T11:52:09.596 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112453.log 2023-12-16T11:52:09.596 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95687.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95687.log.gz 2023-12-16T11:52:09.597 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131256.log 2023-12-16T11:52:09.599 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112453.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112453.log.gz 2023-12-16T11:52:09.600 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107481.log 2023-12-16T11:52:09.600 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131256.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131256.log.gz 2023-12-16T11:52:09.600 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128085.log 2023-12-16T11:52:09.600 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107481.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107481.log.gz 2023-12-16T11:52:09.602 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100935.log 2023-12-16T11:52:09.602 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128085.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128085.log.gz 2023-12-16T11:52:09.602 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98960.log 2023-12-16T11:52:09.604 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100935.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100935.log.gz 2023-12-16T11:52:09.604 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133608.log 2023-12-16T11:52:09.608 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98960.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104272.log.gz 2023-12-16T11:52:09.609 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98960.log.gz 2023-12-16T11:52:09.609 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128065.log 2023-12-16T11:52:09.609 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133608.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133608.log.gz 2023-12-16T11:52:09.621 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130248.log 2023-12-16T11:52:09.623 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128065.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128065.log.gz 2023-12-16T11:52:09.623 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115901.log 2023-12-16T11:52:09.623 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130248.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130248.log.gz 2023-12-16T11:52:09.623 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95471.log 2023-12-16T11:52:09.624 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115901.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115901.log.gz 2023-12-16T11:52:09.624 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95711.log 2023-12-16T11:52:09.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119057.log 2023-12-16T11:52:09.639 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95711.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95711.log.gz 2023-12-16T11:52:09.645 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118056.log 2023-12-16T11:52:09.660 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118664.log 2023-12-16T11:52:09.661 INFO:teuthology.orchestra.run.smithi160.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119057.log.gz 2023-12-16T11:52:09.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118056.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118056.log.gz 2023-12-16T11:52:09.672 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135165.log 2023-12-16T11:52:09.673 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118664.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118664.log.gz 2023-12-16T11:52:09.684 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128847.log 2023-12-16T11:52:09.684 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135165.log.gz 2023-12-16T11:52:09.694 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137979.log 2023-12-16T11:52:09.694 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128847.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128847.log.gz 2023-12-16T11:52:09.707 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112067.log 2023-12-16T11:52:09.708 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137979.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137979.log.gz 2023-12-16T11:52:09.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93591.log 2023-12-16T11:52:09.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.128408.log 2023-12-16T11:52:09.727 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93591.log: 88.6% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101556.log.gz 2023-12-16T11:52:09.728 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126149.log 2023-12-16T11:52:09.730 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.3.128408.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112067.log.gz 2023-12-16T11:52:09.732 INFO:teuthology.orchestra.run.smithi160.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.128408.log.gz 2023-12-16T11:52:09.732 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108504.log 2023-12-16T11:52:09.733 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126149.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126149.log.gz 2023-12-16T11:52:09.733 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108838.log 2023-12-16T11:52:09.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108504.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108504.log.gz 2023-12-16T11:52:09.734 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130710.log 2023-12-16T11:52:09.738 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108838.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108838.log.gz 2023-12-16T11:52:09.738 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120987.log 2023-12-16T11:52:09.738 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130710.log.gz 2023-12-16T11:52:09.738 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98537.log 2023-12-16T11:52:09.738 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120987.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120987.log.gz 2023-12-16T11:52:09.740 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129008.log 2023-12-16T11:52:09.741 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98537.log: -- replaced with /var/log/ceph/cluster2-client.mirror.93591.log.gz 2023-12-16T11:52:09.741 INFO:teuthology.orchestra.run.smithi160.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98537.log.gz 2023-12-16T11:52:09.741 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117557.log 2023-12-16T11:52:09.743 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129008.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129008.log.gz 2023-12-16T11:52:09.743 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116934.log 2023-12-16T11:52:09.743 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117557.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117557.log.gz 2023-12-16T11:52:09.743 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99648.log 2023-12-16T11:52:09.746 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116934.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116934.log.gz 2023-12-16T11:52:09.746 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131088.log 2023-12-16T11:52:09.752 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99648.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99648.log.gz 2023-12-16T11:52:09.752 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127935.log 2023-12-16T11:52:09.752 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131088.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131088.log.gz 2023-12-16T11:52:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120447.log 2023-12-16T11:52:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.127935.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127935.log.gz 2023-12-16T11:52:09.754 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85830.log 2023-12-16T11:52:09.756 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120447.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120447.log.gz 2023-12-16T11:52:09.756 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127103.log 2023-12-16T11:52:09.756 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.85830.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85830.log.gz 2023-12-16T11:52:09.756 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109408.log 2023-12-16T11:52:09.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127103.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127103.log.gz 2023-12-16T11:52:09.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105719.log 2023-12-16T11:52:09.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109408.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109408.log.gz 2023-12-16T11:52:09.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106157.log 2023-12-16T11:52:09.760 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105719.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.105719.log.gz 2023-12-16T11:52:09.761 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117702.log 2023-12-16T11:52:09.761 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106157.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106157.log.gz 2023-12-16T11:52:09.764 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138021.log 2023-12-16T11:52:09.764 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117702.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117702.log.gz 2023-12-16T11:52:09.764 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139785.log 2023-12-16T11:52:09.764 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138021.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138021.log.gz 2023-12-16T11:52:09.764 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116060.log 2023-12-16T11:52:09.764 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139785.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139785.log.gz 2023-12-16T11:52:09.766 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121662.log 2023-12-16T11:52:09.766 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116060.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116060.log.gz 2023-12-16T11:52:09.766 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91564.log 2023-12-16T11:52:09.767 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121662.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121662.log.gz 2023-12-16T11:52:09.767 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94808.log 2023-12-16T11:52:09.785 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.91564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109256.log 2023-12-16T11:52:09.786 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94808.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94808.log.gz 2023-12-16T11:52:09.787 INFO:teuthology.orchestra.run.smithi160.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91564.log.gz 2023-12-16T11:52:09.788 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117139.log 2023-12-16T11:52:09.790 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109256.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109256.log.gz 2023-12-16T11:52:09.790 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94693.log 2023-12-16T11:52:09.793 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117139.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117139.log.gz 2023-12-16T11:52:09.793 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103352.log 2023-12-16T11:52:09.793 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94693.log: 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94693.log.gz 2023-12-16T11:52:09.795 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127062.log 2023-12-16T11:52:09.795 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103352.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.103352.log.gz 2023-12-16T11:52:09.795 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135921.log 2023-12-16T11:52:09.796 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127062.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127062.log.gz 2023-12-16T11:52:09.796 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108331.log 2023-12-16T11:52:09.796 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135921.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135921.log.gz 2023-12-16T11:52:09.798 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126626.log 2023-12-16T11:52:09.798 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108331.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108331.log.gz 2023-12-16T11:52:09.799 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94989.log 2023-12-16T11:52:09.799 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126626.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126626.log.gz 2023-12-16T11:52:09.800 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126108.log 2023-12-16T11:52:09.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94989.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103655.log.gz 2023-12-16T11:52:09.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129743.log 2023-12-16T11:52:09.804 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126108.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126108.log.gz 2023-12-16T11:52:09.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112388.log 2023-12-16T11:52:09.812 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129743.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129743.log.gz 2023-12-16T11:52:09.812 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130542.log 2023-12-16T11:52:09.812 INFO:teuthology.orchestra.run.smithi160.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94989.log.gz/var/log/ceph/cluster1-client.mirror.112388.log: 2023-12-16T11:52:09.812 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136803.log 2023-12-16T11:52:09.812 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112388.log.gz/var/log/ceph/cluster2-client.admin.130542.log: 2023-12-16T11:52:09.812 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130542.log.gz 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109187.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.136803.log: -- /var/log/ceph/cluster2-client.mirror.126170.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136803.log.gz 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127226.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109187.log: /var/log/ceph/cluster2-client.mirror.126170.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109187.log.gz 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126170.log.gz 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125843.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126605.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127226.log.gz/var/log/ceph/cluster2-client.mirror.125843.log: 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103115.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125843.log.gz/var/log/ceph/cluster2-client.mirror.126605.log: 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126605.log.gz 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/cluster2-client.admin.130290.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133776.log 2023-12-16T11:52:09.813 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103115.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103115.log.gz/var/log/ceph/cluster2-client.admin.130290.log: 2023-12-16T11:52:09.814 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130290.log.gz 2023-12-16T11:52:09.814 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91640.log 2023-12-16T11:52:09.814 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133776.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104365.log 2023-12-16T11:52:09.814 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.133776.log.gz 2023-12-16T11:52:09.814 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.91640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109497.log 2023-12-16T11:52:09.815 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104365.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104365.log.gz 2023-12-16T11:52:09.815 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116548.log 2023-12-16T11:52:09.816 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109497.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109497.log.gz 2023-12-16T11:52:09.816 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126129.log 2023-12-16T11:52:09.817 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116548.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116548.log.gz 2023-12-16T11:52:09.818 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121775.log 2023-12-16T11:52:09.818 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126129.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102297.log.gz 2023-12-16T11:52:09.818 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126129.log.gz 2023-12-16T11:52:09.818 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95471.log.gz 2023-12-16T11:52:09.818 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94828.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.121775.log: --verbose -- /var/log/ceph/cluster2-client.mirror.127686.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121775.log.gz 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134619.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94828.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106894.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127686.log: /var/log/ceph/cluster2-client.admin.134619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127665.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134619.log.gz 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127686.log.gz 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130500.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127665.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134871.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106894.log.gz 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130500.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102720.log 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.130500.log.gz 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127665.log.gz 2023-12-16T11:52:09.827 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98121.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134871.log: 0.0%/var/log/ceph/cluster2-client.mirror.102720.log: gzip -- replaced with /var/log/ceph/cluster2-client.admin.134871.log.gz -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112412.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102720.log.gz 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130164.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98121.log: /var/log/ceph/cluster1-client.mirror.112412.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113425.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130164.log: 0.0% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112412.log.gz 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.130164.log.gz 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115455.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126869.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113425.log: /var/log/ceph/cluster2-client.mirror.115455.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113425.log.gz 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115455.log.gz 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122652.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108147.log 2023-12-16T11:52:09.828 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126869.log.gz/var/log/ceph/cluster1-client.mirror.122652.log: 2023-12-16T11:52:09.829 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122652.log.gz 2023-12-16T11:52:09.829 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138567.log 2023-12-16T11:52:09.829 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96039.log 2023-12-16T11:52:09.829 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138567.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138567.log.gz 2023-12-16T11:52:09.829 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134913.log 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96039.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98121.log.gz 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96039.log.gz 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.108147.log.gz 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105892.log 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98667.log 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134913.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134913.log.gz 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127562.log 2023-12-16T11:52:09.840 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105892.log: /var/log/ceph/cluster2-client.mirror.98667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129668.log 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105892.log.gz 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.127562.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.98667.log.gz 86.9% 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110818.log 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.127562.log.gz 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/cluster1-client.mirror.129668.log: -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.mirror.124818.log 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.129668.log.gz 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127206.log 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.124818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135291.log 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124818.log.gz 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104682.log 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135291.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135291.log.gz 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127206.log.gz 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110818.log.gz 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118188.log 2023-12-16T11:52:09.841 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104682.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104682.log.gz 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120297.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/cluster2-client.mirror.118188.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115925.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120297.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.118188.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94206.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120297.log.gz 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117350.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.94206.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115925.log.gz 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134997.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92589.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117350.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.94206.log.gz 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134997.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134997.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140121.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94532.log 2023-12-16T11:52:09.842 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92589.log: /var/log/ceph/cluster2-client.admin.140121.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140121.log.gz 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.92589.log.gz 2023-12-16T11:52:09.843 INFO:teuthology.orchestra.run.smithi160.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117350.log.gz 2023-12-16T11:52:09.843 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.843 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99453.log 2023-12-16T11:52:09.843 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115527.log 2023-12-16T11:52:09.843 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.96555.log 2023-12-16T11:52:09.843 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115527.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115527.log.gz 2023-12-16T11:52:09.843 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115479.log 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.2.96555.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.94532.log.gz 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94554.log 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99453.log.gz 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114172.log 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115479.log.gz 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125252.log 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114172.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94554.log.gz 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114172.log.gz 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94002.log 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106870.log 2023-12-16T11:52:09.850 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125252.log: /var/log/ceph/cluster1-client.mirror.94002.log: 74.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94002.log.gz 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125252.log.gz 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135123.log 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98645.log 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106870.log.gz 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135123.log.gz 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114675.log 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122751.log 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114675.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114675.log.gz 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128659.log 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.122751.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122751.log.gz 2023-12-16T11:52:09.851 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119511.log 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128659.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128659.log.gz 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112111.log 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119511.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98645.log.gz 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95619.log 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112111.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112111.log.gz 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119511.log.gz 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93535.log 2023-12-16T11:52:09.857 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91176.log 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.93535.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95619.log.gz 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.93535.log.gz 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114389.log 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.91176.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113228.log 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.91176.log.gz 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137223.log 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114389.log: /var/log/ceph/cluster1-client.mirror.113228.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114389.log.gz 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113228.log.gz 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102463.log 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136173.log 2023-12-16T11:52:09.858 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137223.log.gz 2023-12-16T11:52:09.877 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99066.log 2023-12-16T11:52:09.878 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136173.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136173.log.gz 2023-12-16T11:52:09.890 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120942.log 2023-12-16T11:52:09.897 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99066.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99066.log.gz 2023-12-16T11:52:09.897 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95156.log 2023-12-16T11:52:09.897 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120942.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120942.log.gz 2023-12-16T11:52:09.899 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112475.log 2023-12-16T11:52:09.899 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95156.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.95156.log.gz 2023-12-16T11:52:09.899 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114847.log 2023-12-16T11:52:09.900 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112475.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112475.log.gz 2023-12-16T11:52:09.900 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116018.log 2023-12-16T11:52:09.901 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114847.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114847.log.gz 2023-12-16T11:52:09.901 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133398.log 2023-12-16T11:52:09.903 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116018.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116018.log.gz 2023-12-16T11:52:09.904 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96333.log 2023-12-16T11:52:09.905 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133398.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133398.log.gz 2023-12-16T11:52:09.905 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114582.log 2023-12-16T11:52:09.905 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96333.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96333.log.gz 2023-12-16T11:52:09.906 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137769.log 2023-12-16T11:52:09.907 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114582.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114582.log.gz 2023-12-16T11:52:09.907 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129508.log 2023-12-16T11:52:09.908 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137769.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137769.log.gz 2023-12-16T11:52:09.908 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128271.log 2023-12-16T11:52:09.909 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129508.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129508.log.gz 2023-12-16T11:52:09.910 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104444.log 2023-12-16T11:52:09.911 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128271.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128271.log.gz 2023-12-16T11:52:09.911 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110500.log 2023-12-16T11:52:09.912 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104444.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.104444.log.gz 2023-12-16T11:52:09.912 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139995.log 2023-12-16T11:52:09.914 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110500.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110500.log.gz 2023-12-16T11:52:09.914 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107252.log 2023-12-16T11:52:09.914 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139995.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139995.log.gz 2023-12-16T11:52:09.915 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126708.log 2023-12-16T11:52:09.915 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107252.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107252.log.gz 2023-12-16T11:52:09.916 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97484.log 2023-12-16T11:52:09.917 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126708.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126708.log.gz 2023-12-16T11:52:09.917 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125678.log 2023-12-16T11:52:09.918 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97484.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97484.log.gz 2023-12-16T11:52:09.918 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101747.log 2023-12-16T11:52:09.919 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125678.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125678.log.gz 2023-12-16T11:52:09.920 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103517.log 2023-12-16T11:52:09.920 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101747.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101747.log.gz 2023-12-16T11:52:09.921 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110898.log 2023-12-16T11:52:09.921 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103517.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103517.log.gz 2023-12-16T11:52:09.922 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128023.log 2023-12-16T11:52:09.923 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110898.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110898.log.gz 2023-12-16T11:52:09.924 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118829.log 2023-12-16T11:52:09.925 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.128023.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128023.log.gz 2023-12-16T11:52:09.925 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91470.log 2023-12-16T11:52:09.926 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118829.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118829.log.gz 2023-12-16T11:52:09.928 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121415.log 2023-12-16T11:52:09.928 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.91470.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91470.log.gz 2023-12-16T11:52:09.928 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100177.log 2023-12-16T11:52:09.929 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121415.log: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121415.log.gz 2023-12-16T11:52:09.929 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124626.log 2023-12-16T11:52:09.929 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100348.log.gz 2023-12-16T11:52:09.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108460.log 2023-12-16T11:52:09.931 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124626.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100177.log.gz 2023-12-16T11:52:09.931 INFO:teuthology.orchestra.run.smithi160.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124626.log.gz 2023-12-16T11:52:09.931 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108418.log 2023-12-16T11:52:09.931 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103389.log 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108460.log: /var/log/ceph/cluster2-client.mirror.108418.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108460.log.gz 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108418.log.gz 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127267.log 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.103389.log: -- /var/log/ceph/cluster2-client.mirror.92606.log 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103389.log.gz 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129365.log 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92606.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92606.log.gz 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92674.log 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129365.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127267.log.gz 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129365.log.gz 2023-12-16T11:52:09.935 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98465.log 2023-12-16T11:52:09.937 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101185.log 2023-12-16T11:52:09.937 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92674.log: /var/log/ceph/cluster1-client.mirror.98465.log: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92674.log.gz 2023-12-16T11:52:09.937 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107348.log 2023-12-16T11:52:09.937 INFO:teuthology.orchestra.run.smithi160.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98465.log.gz 2023-12-16T11:52:09.937 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101885.log 2023-12-16T11:52:09.937 INFO:teuthology.orchestra.run.smithi160.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101185.log.gz 2023-12-16T11:52:09.937 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107348.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107348.log.gz 2023-12-16T11:52:09.939 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129718.log 2023-12-16T11:52:09.939 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.101885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112633.log 2023-12-16T11:52:09.939 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129718.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129718.log.gz 2023-12-16T11:52:09.939 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116664.log 2023-12-16T11:52:09.939 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112633.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112633.log.gz 2023-12-16T11:52:09.940 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116126.log 2023-12-16T11:52:09.940 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116664.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.116664.log.gz 2023-12-16T11:52:09.941 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117994.log 2023-12-16T11:52:09.941 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116126.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116126.log.gz 2023-12-16T11:52:09.942 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111930.log 2023-12-16T11:52:09.942 INFO:teuthology.orchestra.run.smithi160.stderr: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101885.log.gz 2023-12-16T11:52:09.945 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109477.log 2023-12-16T11:52:09.945 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111930.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117994.log.gz 2023-12-16T11:52:09.945 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112340.log 2023-12-16T11:52:09.945 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111930.log.gz 2023-12-16T11:52:09.945 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110091.log 2023-12-16T11:52:09.945 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112340.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112340.log.gz 2023-12-16T11:52:09.945 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126375.log 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110091.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110091.log.gz 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109477.log.gz 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125483.log 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111954.log 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126375.log.gz 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112810.log 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111954.log: 87.7% 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111954.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.125483.log.gz 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131844.log 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128970.log 2023-12-16T11:52:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112810.log: /var/log/ceph/cluster2-client.admin.131844.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131844.log.gz 2023-12-16T11:52:09.950 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112810.log.gz 2023-12-16T11:52:09.950 INFO:teuthology.orchestra.run.smithi160.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.96555.log.gz 2023-12-16T11:52:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115052.log 2023-12-16T11:52:09.954 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.128970.log: -- /var/log/ceph/cluster1-client.mirror.99367.log 2023-12-16T11:52:09.954 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132096.log 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115052.log: /var/log/ceph/cluster1-client.mirror.99367.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115052.log.gz 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128970.log.gz 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128147.log 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116331.log 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132096.log.gz 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104385.log 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128147.log.gz 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106822.log 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104385.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116331.log.gz 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104385.log.gz 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100977.log 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125901.log 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100977.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106822.log.gz 2023-12-16T11:52:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99668.log 2023-12-16T11:52:09.957 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125901.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125901.log.gz 2023-12-16T11:52:09.957 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136509.log 2023-12-16T11:52:09.957 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99668.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99367.log.gz 2023-12-16T11:52:09.957 INFO:teuthology.orchestra.run.smithi160.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91640.log.gz 2023-12-16T11:52:09.957 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120650.log 2023-12-16T11:52:09.958 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136509.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136509.log.gz 2023-12-16T11:52:09.958 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109786.log 2023-12-16T11:52:09.958 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98937.log 2023-12-16T11:52:09.958 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120650.log: /var/log/ceph/cluster2-client.mirror.109786.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120650.log.gz 2023-12-16T11:52:09.958 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99238.log 2023-12-16T11:52:09.961 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98937.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99668.log.gz 2023-12-16T11:52:09.961 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106244.log 2023-12-16T11:52:09.964 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99238.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98937.log.gz 2023-12-16T11:52:09.964 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90500.log 2023-12-16T11:52:09.965 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106244.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106244.log.gz 2023-12-16T11:52:09.966 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118788.log 2023-12-16T11:52:09.966 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90500.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90500.log.gz 2023-12-16T11:52:09.966 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96994.log 2023-12-16T11:52:09.967 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118788.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99238.log.gz 87.5% 2023-12-16T11:52:09.967 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118788.log.gz 2023-12-16T11:52:09.967 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102112.log 2023-12-16T11:52:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125439.log 2023-12-16T11:52:09.968 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96994.log: /var/log/ceph/cluster2-client.mirror.102112.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96994.log.gz 2023-12-16T11:52:09.969 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96378.log 2023-12-16T11:52:09.969 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125439.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125439.log.gz 2023-12-16T11:52:09.970 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110773.log 2023-12-16T11:52:09.975 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96378.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96378.log.gz 2023-12-16T11:52:09.976 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126728.log 2023-12-16T11:52:09.976 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110773.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110773.log.gz 2023-12-16T11:52:09.977 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111460.log 2023-12-16T11:52:09.977 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126728.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126728.log.gz 2023-12-16T11:52:09.978 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106222.log 2023-12-16T11:52:09.978 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111460.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111460.log.gz 2023-12-16T11:52:09.979 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112879.log 2023-12-16T11:52:09.981 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106222.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106222.log.gz 2023-12-16T11:52:09.981 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137811.log 2023-12-16T11:52:09.981 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109786.log.gz 2023-12-16T11:52:09.983 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112879.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112879.log.gz 2023-12-16T11:52:09.984 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104464.log 2023-12-16T11:52:09.984 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137811.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105136.log 2023-12-16T11:52:09.985 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.137811.log.gz 2023-12-16T11:52:09.985 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104857.log 2023-12-16T11:52:09.985 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104464.log: /var/log/ceph/cluster2-client.mirror.105136.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104464.log.gz 2023-12-16T11:52:09.985 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118726.log 2023-12-16T11:52:09.986 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104857.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104857.log.gz 2023-12-16T11:52:09.986 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96421.log 2023-12-16T11:52:09.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118726.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118726.log.gz 2023-12-16T11:52:09.987 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102994.log 2023-12-16T11:52:09.993 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96421.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96421.log.gz 2023-12-16T11:52:09.993 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126750.log 2023-12-16T11:52:09.994 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102994.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102994.log.gz 2023-12-16T11:52:09.994 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118014.log 2023-12-16T11:52:09.995 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126750.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126750.log.gz 2023-12-16T11:52:09.995 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92178.log 2023-12-16T11:52:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118014.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118014.log.gz 2023-12-16T11:52:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116399.log 2023-12-16T11:52:09.997 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.92178.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92178.log.gz 2023-12-16T11:52:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95295.log 2023-12-16T11:52:09.998 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116399.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116399.log.gz 2023-12-16T11:52:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-12-16T11:52:09.999 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95295.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95295.log.gz 2023-12-16T11:52:10.000 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109235.log 2023-12-16T11:52:10.004 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2.audit.log: 92.8% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-12-16T11:52:10.004 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135669.log 2023-12-16T11:52:10.005 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109235.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109235.log.gz 2023-12-16T11:52:10.005 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109119.log 2023-12-16T11:52:10.006 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135669.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135669.log.gz 2023-12-16T11:52:10.006 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136635.log 2023-12-16T11:52:10.007 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109119.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109119.log.gz 2023-12-16T11:52:10.007 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100897.log 2023-12-16T11:52:10.007 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136635.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136635.log.gz 2023-12-16T11:52:10.008 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107186.log 2023-12-16T11:52:10.020 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104620.log 2023-12-16T11:52:10.021 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107186.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107186.log.gz 2023-12-16T11:52:10.035 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122734.log 2023-12-16T11:52:10.048 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104620.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104620.log.gz 2023-12-16T11:52:10.054 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133440.log 2023-12-16T11:52:10.054 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.122734.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122734.log.gz 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105534.log 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133440.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133440.log.gz 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102463.log.gz 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131382.log 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105534.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105534.log.gz 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134577.log 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131382.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131382.log.gz 2023-12-16T11:52:10.070 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108725.log 2023-12-16T11:52:10.071 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134577.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134577.log.gz 2023-12-16T11:52:10.071 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103174.log 2023-12-16T11:52:10.071 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108725.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108725.log.gz 2023-12-16T11:52:10.071 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122699.log 2023-12-16T11:52:10.079 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99582.log 2023-12-16T11:52:10.079 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122699.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122699.log.gz 2023-12-16T11:52:10.093 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119263.log 2023-12-16T11:52:10.098 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99582.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99582.log.gz 2023-12-16T11:52:10.100 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117094.log 2023-12-16T11:52:10.106 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119263.log: 93.7% 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119263.log.gz 2023-12-16T11:52:10.106 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.93964.log.gz 2023-12-16T11:52:10.108 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99089.log 2023-12-16T11:52:10.108 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.117094.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.117094.log.gz 2023-12-16T11:52:10.108 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117412.log 2023-12-16T11:52:10.114 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99089.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99089.log.gz 2023-12-16T11:52:10.114 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119408.log 2023-12-16T11:52:10.117 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117412.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117412.log.gz 2023-12-16T11:52:10.117 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103016.log 2023-12-16T11:52:10.117 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119408.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119408.log.gz 2023-12-16T11:52:10.117 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103053.log 2023-12-16T11:52:10.142 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121077.log 2023-12-16T11:52:10.142 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103053.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103053.log.gz 2023-12-16T11:52:10.143 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127706.log 2023-12-16T11:52:10.143 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121077.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121077.log.gz 2023-12-16T11:52:10.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98338.log 2023-12-16T11:52:10.153 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127706.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127706.log.gz 2023-12-16T11:52:10.164 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110838.log 2023-12-16T11:52:10.164 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98338.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98338.log.gz 2023-12-16T11:52:10.178 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96309.log 2023-12-16T11:52:10.179 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110838.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110838.log.gz 2023-12-16T11:52:10.193 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126412.log 2023-12-16T11:52:10.193 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96309.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96309.log.gz 2023-12-16T11:52:10.212 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119098.log 2023-12-16T11:52:10.213 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126412.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126412.log.gz 2023-12-16T11:52:10.221 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108922.log 2023-12-16T11:52:10.232 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119098.log: 93.7% 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119098.log.gz 2023-12-16T11:52:10.232 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100897.log.gz 2023-12-16T11:52:10.232 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100289.log 2023-12-16T11:52:10.232 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108922.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108922.log.gz 2023-12-16T11:52:10.232 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111609.log 2023-12-16T11:52:10.232 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100289.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100289.log.gz 2023-12-16T11:52:10.235 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105094.log 2023-12-16T11:52:10.235 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111609.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111609.log.gz 2023-12-16T11:52:10.235 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119470.log 2023-12-16T11:52:10.235 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105094.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105094.log.gz 2023-12-16T11:52:10.235 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117764.log 2023-12-16T11:52:10.236 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119470.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119470.log.gz 2023-12-16T11:52:10.236 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115435.log 2023-12-16T11:52:10.238 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117764.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117764.log.gz 2023-12-16T11:52:10.238 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100241.log 2023-12-16T11:52:10.238 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115435.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115435.log.gz 2023-12-16T11:52:10.239 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92110.log 2023-12-16T11:52:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.100241.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100241.log.gz 2023-12-16T11:52:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133944.log 2023-12-16T11:52:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.92110.log: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92110.log.gz 2023-12-16T11:52:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95343.log 2023-12-16T11:52:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133944.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133944.log.gz 2023-12-16T11:52:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138315.log 2023-12-16T11:52:10.245 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95343.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95343.log.gz 2023-12-16T11:52:10.245 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.96102.log 2023-12-16T11:52:10.245 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138315.log.gz 2023-12-16T11:52:10.245 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120965.log 2023-12-16T11:52:10.248 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.1.96102.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.96102.log.gz 2023-12-16T11:52:10.248 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96065.log 2023-12-16T11:52:10.249 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120965.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120965.log.gz 2023-12-16T11:52:10.249 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110323.log 2023-12-16T11:52:10.250 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96065.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96065.log.gz 2023-12-16T11:52:10.251 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136761.log 2023-12-16T11:52:10.252 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110323.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110323.log.gz 2023-12-16T11:52:10.252 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115672.log 2023-12-16T11:52:10.252 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136761.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136761.log.gz 2023-12-16T11:52:10.253 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126478.log 2023-12-16T11:52:10.253 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115672.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115672.log.gz 2023-12-16T11:52:10.254 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108259.log 2023-12-16T11:52:10.255 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126478.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126478.log.gz 2023-12-16T11:52:10.255 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126646.log 2023-12-16T11:52:10.256 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108259.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108259.log.gz 2023-12-16T11:52:10.257 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114606.log 2023-12-16T11:52:10.258 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126646.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126646.log.gz 2023-12-16T11:52:10.258 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121257.log 2023-12-16T11:52:10.258 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114606.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114606.log.gz 2023-12-16T11:52:10.260 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122337.log 2023-12-16T11:52:10.260 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121257.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121257.log.gz 2023-12-16T11:52:10.261 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126956.log 2023-12-16T11:52:10.261 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122337.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122337.log.gz 2023-12-16T11:52:10.262 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139659.log 2023-12-16T11:52:10.263 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126956.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126956.log.gz 2023-12-16T11:52:10.263 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119821.log 2023-12-16T11:52:10.263 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139659.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139659.log.gz 2023-12-16T11:52:10.264 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106412.log 2023-12-16T11:52:10.266 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119821.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119821.log.gz 2023-12-16T11:52:10.266 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112296.log 2023-12-16T11:52:10.267 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106412.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106412.log.gz 2023-12-16T11:52:10.267 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114984.log 2023-12-16T11:52:10.269 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112296.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112296.log.gz 2023-12-16T11:52:10.269 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97018.log 2023-12-16T11:52:10.270 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114984.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114984.log.gz 2023-12-16T11:52:10.272 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114695.log 2023-12-16T11:52:10.274 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97018.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.97018.log.gz 2023-12-16T11:52:10.274 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140373.log 2023-12-16T11:52:10.274 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114695.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114695.log.gz 2023-12-16T11:52:10.274 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135837.log 2023-12-16T11:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140373.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140373.log.gz 2023-12-16T11:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116821.log 2023-12-16T11:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135837.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135837.log.gz 2023-12-16T11:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118767.log 2023-12-16T11:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116821.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116821.log.gz 2023-12-16T11:52:10.278 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94328.log 2023-12-16T11:52:10.280 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118767.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118767.log.gz 2023-12-16T11:52:10.280 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119697.log 2023-12-16T11:52:10.281 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94328.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.94328.log.gz 2023-12-16T11:52:10.281 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107554.log 2023-12-16T11:52:10.283 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119697.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119697.log.gz 2023-12-16T11:52:10.283 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105173.log 2023-12-16T11:52:10.284 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107554.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107554.log.gz 2023-12-16T11:52:10.285 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101769.log 2023-12-16T11:52:10.286 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105173.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105173.log.gz 2023-12-16T11:52:10.286 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104230.log 2023-12-16T11:52:10.295 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101769.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102112.log.gz 2023-12-16T11:52:10.296 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124420.log 2023-12-16T11:52:10.297 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104230.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104230.log.gz 2023-12-16T11:52:10.297 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120897.log 2023-12-16T11:52:10.299 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124420.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124420.log.gz 2023-12-16T11:52:10.299 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120492.log 2023-12-16T11:52:10.299 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120897.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120897.log.gz 2023-12-16T11:52:10.299 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119594.log 2023-12-16T11:52:10.302 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120492.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120492.log.gz 2023-12-16T11:52:10.302 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108946.log 2023-12-16T11:52:10.302 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119594.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119594.log.gz 2023-12-16T11:52:10.302 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113521.log 2023-12-16T11:52:10.303 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108946.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108946.log.gz 2023-12-16T11:52:10.304 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98851.log 2023-12-16T11:52:10.304 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113521.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113521.log.gz 2023-12-16T11:52:10.305 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120214.log 2023-12-16T11:52:10.310 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98851.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98851.log.gz 2023-12-16T11:52:10.310 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95735.log 2023-12-16T11:52:10.312 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120214.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120214.log.gz 2023-12-16T11:52:10.312 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126891.log 2023-12-16T11:52:10.313 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95735.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95735.log.gz 2023-12-16T11:52:10.313 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104506.log 2023-12-16T11:52:10.327 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127788.log 2023-12-16T11:52:10.327 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104506.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104506.log.gz 2023-12-16T11:52:10.341 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132684.log 2023-12-16T11:52:10.342 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127788.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.127788.log.gz 2023-12-16T11:52:10.342 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118974.log 2023-12-16T11:52:10.343 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132684.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132684.log.gz 2023-12-16T11:52:10.354 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132516.log 2023-12-16T11:52:10.354 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118974.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118974.log.gz 2023-12-16T11:52:10.363 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.97550.log 2023-12-16T11:52:10.363 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132516.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132516.log.gz 2023-12-16T11:52:10.372 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104719.log 2023-12-16T11:52:10.389 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.97550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118105.log 2023-12-16T11:52:10.389 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104719.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104719.log.gz 2023-12-16T11:52:10.390 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103174.log.gz 2023-12-16T11:52:10.393 INFO:teuthology.orchestra.run.smithi160.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.97550.log.gz 2023-12-16T11:52:10.393 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129050.log 2023-12-16T11:52:10.393 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118105.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118105.log.gz 2023-12-16T11:52:10.394 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139869.log 2023-12-16T11:52:10.403 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129050.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105136.log.gz 2023-12-16T11:52:10.405 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94512.log 2023-12-16T11:52:10.405 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139869.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139869.log.gz 2023-12-16T11:52:10.405 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104840.log 2023-12-16T11:52:10.411 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94512.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94512.log.gz 2023-12-16T11:52:10.411 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131676.log 2023-12-16T11:52:10.412 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104840.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104840.log.gz 2023-12-16T11:52:10.415 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130962.log 2023-12-16T11:52:10.415 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131676.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131676.log.gz 2023-12-16T11:52:10.415 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120762.log 2023-12-16T11:52:10.415 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130962.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130962.log.gz 2023-12-16T11:52:10.415 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111975.log 2023-12-16T11:52:10.415 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120762.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120762.log.gz 2023-12-16T11:52:10.415 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117118.log 2023-12-16T11:52:10.417 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111975.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111975.log.gz 2023-12-16T11:52:10.417 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130626.log 2023-12-16T11:52:10.419 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.117118.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.117118.log.gz 2023-12-16T11:52:10.419 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99831.log 2023-12-16T11:52:10.419 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130626.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130626.log.gz 2023-12-16T11:52:10.420 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107300.log 2023-12-16T11:52:10.421 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99831.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99831.log.gz 2023-12-16T11:52:10.421 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124544.log 2023-12-16T11:52:10.422 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107300.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107300.log.gz 2023-12-16T11:52:10.423 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111585.log 2023-12-16T11:52:10.424 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124544.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124544.log.gz 2023-12-16T11:52:10.424 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124249.log 2023-12-16T11:52:10.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111585.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111585.log.gz 2023-12-16T11:52:10.425 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112517.log 2023-12-16T11:52:10.426 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.124249.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124249.log.gz 2023-12-16T11:52:10.426 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135375.log 2023-12-16T11:52:10.427 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112517.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112517.log.gz 2023-12-16T11:52:10.429 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129693.log 2023-12-16T11:52:10.429 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135375.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135375.log.gz 2023-12-16T11:52:10.429 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95384.log 2023-12-16T11:52:10.430 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129693.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129693.log.gz 2023-12-16T11:52:10.430 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115768.log 2023-12-16T11:52:10.436 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95384.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95384.log.gz 2023-12-16T11:52:10.436 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115881.log 2023-12-16T11:52:10.437 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115768.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115768.log.gz 2023-12-16T11:52:10.437 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102700.log 2023-12-16T11:52:10.439 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115881.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115881.log.gz 2023-12-16T11:52:10.439 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122472.log 2023-12-16T11:52:10.445 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102700.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100977.log.gz 2023-12-16T11:52:10.452 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118208.log 2023-12-16T11:52:10.452 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122472.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122472.log.gz 2023-12-16T11:52:10.453 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112654.log 2023-12-16T11:52:10.455 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118208.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118208.log.gz 2023-12-16T11:52:10.456 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140079.log 2023-12-16T11:52:10.458 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112654.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112654.log.gz 2023-12-16T11:52:10.458 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102678.log 2023-12-16T11:52:10.459 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140079.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140079.log.gz 2023-12-16T11:52:10.459 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133482.log 2023-12-16T11:52:10.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102678.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102678.log.gz 2023-12-16T11:52:10.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98894.log 2023-12-16T11:52:10.462 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133482.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133482.log.gz 2023-12-16T11:52:10.462 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91134.log 2023-12-16T11:52:10.468 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98894.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98894.log.gz 2023-12-16T11:52:10.468 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111280.log 2023-12-16T11:52:10.469 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.91134.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91134.log.gz 2023-12-16T11:52:10.469 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121707.log 2023-12-16T11:52:10.471 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111280.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111280.log.gz 2023-12-16T11:52:10.471 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125047.log 2023-12-16T11:52:10.472 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121707.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121707.log.gz 2023-12-16T11:52:10.473 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99911.log 2023-12-16T11:52:10.480 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125047.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125047.log.gz 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132726.log 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99911.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.99911.log.gz 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121482.log 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132726.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132726.log.gz 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115278.log 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121482.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121482.log.gz 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117973.log 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115278.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115278.log.gz 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103692.log 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117973.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117973.log.gz 2023-12-16T11:52:10.481 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100441.log 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103692.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103692.log.gz 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136215.log 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100441.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100441.log.gz 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112087.log 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136215.log.gz 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119987.log 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112087.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112087.log.gz 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115572.log 2023-12-16T11:52:10.486 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119987.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119987.log.gz 2023-12-16T11:52:10.487 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92127.log 2023-12-16T11:52:10.489 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115572.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115572.log.gz 2023-12-16T11:52:10.489 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122112.log 2023-12-16T11:52:10.490 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.92127.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92127.log.gz 2023-12-16T11:52:10.490 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103596.log 2023-12-16T11:52:10.491 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122112.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122112.log.gz 2023-12-16T11:52:10.491 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128745.log 2023-12-16T11:52:10.492 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103596.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103596.log.gz 2023-12-16T11:52:10.492 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118456.log 2023-12-16T11:52:10.493 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128745.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128745.log.gz 2023-12-16T11:52:10.494 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105940.log 2023-12-16T11:52:10.496 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118456.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118456.log.gz 2023-12-16T11:52:10.496 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117330.log 2023-12-16T11:52:10.496 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105940.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.105940.log.gz 2023-12-16T11:52:10.497 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109039.log 2023-12-16T11:52:10.498 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117330.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117330.log.gz 2023-12-16T11:52:10.499 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100855.log 2023-12-16T11:52:10.501 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109039.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.109039.log.gz 2023-12-16T11:52:10.501 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112246.log 2023-12-16T11:52:10.502 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100855.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100855.log.gz 2023-12-16T11:52:10.503 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91684.log 2023-12-16T11:52:10.503 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112246.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112246.log.gz 2023-12-16T11:52:10.504 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123463.log 2023-12-16T11:52:10.505 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.91684.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91684.log.gz 2023-12-16T11:52:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108167.log 2023-12-16T11:52:10.506 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.123463.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.123463.log.gz 2023-12-16T11:52:10.507 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116869.log 2023-12-16T11:52:10.509 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108167.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108167.log.gz 2023-12-16T11:52:10.509 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90336.log 2023-12-16T11:52:10.510 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116869.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116869.log.gz 2023-12-16T11:52:10.511 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101806.log 2023-12-16T11:52:10.511 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90336.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90336.log.gz 2023-12-16T11:52:10.512 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137181.log 2023-12-16T11:52:10.513 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101806.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101806.log.gz 2023-12-16T11:52:10.513 INFO:teuthology.orchestra.run.smithi160.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-12-16T11:52:10.514 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92623.log 2023-12-16T11:52:10.514 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137181.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137181.log.gz 2023-12-16T11:52:10.514 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120875.log 2023-12-16T11:52:10.515 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139197.log 2023-12-16T11:52:10.515 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92623.log: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92623.log.gz 2023-12-16T11:52:10.515 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128230.log 2023-12-16T11:52:10.515 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120875.log.gz 2023-12-16T11:52:10.515 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101769.log.gz 2023-12-16T11:52:10.515 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139197.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139197.log.gz 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133272.log 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103253.log 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128230.log: /var/log/ceph/cluster2-client.admin.133272.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133272.log.gz 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.128230.log.gz 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104778.log 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.103253.log: -- /var/log/ceph/cluster2-client.admin.133902.log 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97660.log 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133902.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133902.log.gz 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104778.log.gz 2023-12-16T11:52:10.518 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105760.log 2023-12-16T11:52:10.519 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103231.log 2023-12-16T11:52:10.520 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97660.log: /var/log/ceph/cluster1-client.mirror.105760.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105760.log.gz 2023-12-16T11:52:10.523 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97660.log.gz 2023-12-16T11:52:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107389.log 2023-12-16T11:52:10.524 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94424.log 2023-12-16T11:52:10.525 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103231.log: /var/log/ceph/cluster1-client.mirror.107389.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103231.log.gz 2023-12-16T11:52:10.525 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90966.log 2023-12-16T11:52:10.526 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94424.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94424.log.gz 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107324.log 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107389.log.gz 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90966.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90966.log.gz 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138273.log 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134409.log 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107324.log.gz 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138273.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138273.log.gz 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132306.log 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134409.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134409.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126791.log 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127887.log 2023-12-16T11:52:10.534 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126791.log: /var/log/ceph/cluster2-client.admin.132306.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132306.log.gz 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133314.log 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.126791.log.gz 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100693.log 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133314.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133314.log.gz 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127887.log.gz 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127867.log 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102441.log 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.100693.log.gz 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91604.log 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127867.log.gz 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117932.log 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.91604.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102441.log.gz 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-12-16T11:52:10.535 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117932.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117932.log.gz 2023-12-16T11:52:10.536 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131214.log 2023-12-16T11:52:10.546 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-osd.2.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103016.log.gz 2023-12-16T11:52:10.547 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95547.log 2023-12-16T11:52:10.548 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131214.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131214.log.gz 2023-12-16T11:52:10.548 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104191.log 2023-12-16T11:52:10.549 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95547.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.95547.log.gz 2023-12-16T11:52:10.549 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109948.log 2023-12-16T11:52:10.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124564.log 2023-12-16T11:52:10.568 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109948.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109948.log.gz 2023-12-16T11:52:10.574 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118995.log 2023-12-16T11:52:10.575 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124564.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126891.log.gz 2023-12-16T11:52:10.576 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124564.log.gz 2023-12-16T11:52:10.587 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101826.log 2023-12-16T11:52:10.588 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118995.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118995.log.gz 2023-12-16T11:52:10.588 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128000.log 2023-12-16T11:52:10.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101826.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101826.log.gz 2023-12-16T11:52:10.591 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95823.log 2023-12-16T11:52:10.591 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.128000.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128000.log.gz 2023-12-16T11:52:10.591 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105964.log 2023-12-16T11:52:10.597 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95823.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95823.log.gz 2023-12-16T11:52:10.597 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85790.log 2023-12-16T11:52:10.598 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105964.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.105964.log.gz 2023-12-16T11:52:10.598 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104640.log 2023-12-16T11:52:10.599 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.85790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85790.log.gz 2023-12-16T11:52:10.601 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136005.log 2023-12-16T11:52:10.601 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104640.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104640.log.gz 2023-12-16T11:52:10.601 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121820.log 2023-12-16T11:52:10.601 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136005.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136005.log.gz 2023-12-16T11:52:10.603 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97723.log 2023-12-16T11:52:10.603 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121820.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121820.log.gz 2023-12-16T11:52:10.603 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104699.log 2023-12-16T11:52:10.608 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97723.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.91604.log.gz 2023-12-16T11:52:10.608 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97723.log.gz 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120338.log 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.104699.log: -- /var/log/ceph/cluster2-client.mirror.98737.log 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120338.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104699.log.gz 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120338.log.gz 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118436.log 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109298.log 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr: 89.3%/var/log/ceph/cluster2-client.mirror.118436.log: -- replaced with /var/log/ceph/cluster2-client.mirror.98737.log.gz 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138693.log 2023-12-16T11:52:10.613 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118436.log.gz 2023-12-16T11:52:10.614 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97841.log 2023-12-16T11:52:10.614 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109298.log.gz 2023-12-16T11:52:10.614 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138693.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138693.log.gz 2023-12-16T11:52:10.614 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95271.log 2023-12-16T11:52:10.614 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125636.log 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97841.log: /var/log/ceph/cluster1-client.mirror.95271.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97841.log.gz 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95271.log.gz 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121527.log 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125636.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125636.log.gz 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124585.log 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137097.log 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121527.log: /var/log/ceph/cluster2-client.mirror.124585.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121527.log.gz 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109384.log 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124585.log.gz 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137097.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137097.log.gz 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92691.log 2023-12-16T11:52:10.618 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111260.log 2023-12-16T11:52:10.619 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109384.log: /var/log/ceph/cluster2-client.mirror.92691.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109384.log.gz 2023-12-16T11:52:10.619 INFO:teuthology.orchestra.run.smithi160.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92691.log.gz 2023-12-16T11:52:10.619 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114538.log 2023-12-16T11:52:10.619 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98980.log 2023-12-16T11:52:10.620 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111260.log: /var/log/ceph/cluster2-client.mirror.114538.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.114538.log.gz 2023-12-16T11:52:10.620 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.111260.log.gz 2023-12-16T11:52:10.620 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103095.log 2023-12-16T11:52:10.620 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96102.log 2023-12-16T11:52:10.625 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98980.log: /var/log/ceph/cluster2-client.mirror.103095.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98980.log.gz 2023-12-16T11:52:10.626 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112983.log 2023-12-16T11:52:10.626 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96102.log: 74.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96102.log.gz 2023-12-16T11:52:10.627 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114341.log 2023-12-16T11:52:10.627 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112983.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112983.log.gz 2023-12-16T11:52:10.628 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119966.log 2023-12-16T11:52:10.628 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114341.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114341.log.gz 2023-12-16T11:52:10.629 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122180.log 2023-12-16T11:52:10.629 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119966.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119966.log.gz 2023-12-16T11:52:10.630 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133692.log 2023-12-16T11:52:10.630 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122180.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122180.log.gz 2023-12-16T11:52:10.631 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99771.log 2023-12-16T11:52:10.631 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133692.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133692.log.gz 2023-12-16T11:52:10.632 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101727.log 2023-12-16T11:52:10.632 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99771.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.99771.log.gz 2023-12-16T11:52:10.633 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93725.log 2023-12-16T11:52:10.633 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101727.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101727.log.gz 2023-12-16T11:52:10.634 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113055.log 2023-12-16T11:52:10.634 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.93725.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93725.log.gz 2023-12-16T11:52:10.635 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.129210.log 2023-12-16T11:52:10.636 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113055.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113055.log.gz 2023-12-16T11:52:10.636 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124337.log 2023-12-16T11:52:10.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.129210.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.129210.log.gz 2023-12-16T11:52:10.638 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121550.log 2023-12-16T11:52:10.638 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.124337.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124337.log.gz 2023-12-16T11:52:10.639 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107578.log 2023-12-16T11:52:10.639 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121550.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121550.log.gz 2023-12-16T11:52:10.640 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106915.log 2023-12-16T11:52:10.640 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107578.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107578.log.gz 2023-12-16T11:52:10.641 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124503.log 2023-12-16T11:52:10.643 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106915.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106915.log.gz 2023-12-16T11:52:10.643 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91344.log 2023-12-16T11:52:10.644 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124503.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124503.log.gz 2023-12-16T11:52:10.644 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137139.log 2023-12-16T11:52:10.645 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.91344.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91344.log.gz 2023-12-16T11:52:10.645 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97102.log 2023-12-16T11:52:10.645 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137139.log.gz 2023-12-16T11:52:10.646 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136341.log 2023-12-16T11:52:10.653 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97102.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97102.log.gz 2023-12-16T11:52:10.653 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96772.log 2023-12-16T11:52:10.653 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136341.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136341.log.gz 2023-12-16T11:52:10.653 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114196.log 2023-12-16T11:52:10.654 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.96772.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.96772.log.gz 2023-12-16T11:52:10.654 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111565.log 2023-12-16T11:52:10.655 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114196.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114196.log.gz 2023-12-16T11:52:10.655 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120405.log 2023-12-16T11:52:10.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111565.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111565.log.gz 2023-12-16T11:52:10.657 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106027.log 2023-12-16T11:52:10.658 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120405.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120405.log.gz 2023-12-16T11:52:10.658 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111014.log 2023-12-16T11:52:10.659 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106027.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106027.log.gz 2023-12-16T11:52:10.659 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132894.log 2023-12-16T11:52:10.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111014.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111014.log.gz 2023-12-16T11:52:10.661 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92281.log 2023-12-16T11:52:10.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132894.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132894.log.gz 2023-12-16T11:52:10.663 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125194.log 2023-12-16T11:52:10.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.92281.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.admin.92281.log.gz 2023-12-16T11:52:10.663 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104820.log 2023-12-16T11:52:10.664 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125194.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125194.log.gz 2023-12-16T11:52:10.664 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105444.log 2023-12-16T11:52:10.678 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92730.log 2023-12-16T11:52:10.679 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105444.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103253.log.gz 2023-12-16T11:52:10.690 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99751.log 2023-12-16T11:52:10.691 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92730.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92730.log.gz 2023-12-16T11:52:10.704 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125781.log 2023-12-16T11:52:10.705 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99751.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99751.log.gz 2023-12-16T11:52:10.708 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105444.log.gz 2023-12-16T11:52:10.708 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110028.log 2023-12-16T11:52:10.709 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125781.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125781.log.gz 2023-12-16T11:52:10.710 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121955.log 2023-12-16T11:52:10.710 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110028.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110028.log.gz 2023-12-16T11:52:10.711 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100199.log 2023-12-16T11:52:10.712 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121955.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121955.log.gz 2023-12-16T11:52:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129210.log 2023-12-16T11:52:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100199.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.100199.log.gz 2023-12-16T11:52:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95247.log 2023-12-16T11:52:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129210.log: 78.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129210.log.gz 2023-12-16T11:52:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119656.log 2023-12-16T11:52:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95247.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95247.log.gz 2023-12-16T11:52:10.716 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96009.log 2023-12-16T11:52:10.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119656.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119656.log.gz 2023-12-16T11:52:10.718 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100458.log 2023-12-16T11:52:10.718 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.96009.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.96009.log.gz 2023-12-16T11:52:10.719 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120111.log 2023-12-16T11:52:10.720 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100458.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.100458.log.gz 2023-12-16T11:52:10.720 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96802.log 2023-12-16T11:52:10.721 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120111.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120111.log.gz 2023-12-16T11:52:10.721 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105916.log 2023-12-16T11:52:10.722 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.96802.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.96802.log.gz 2023-12-16T11:52:10.722 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110368.log 2023-12-16T11:52:10.724 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.105916.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105916.log.gz 2023-12-16T11:52:10.724 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104603.log 2023-12-16T11:52:10.726 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110368.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110368.log.gz 2023-12-16T11:52:10.726 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138609.log 2023-12-16T11:52:10.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104603.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104603.log.gz 2023-12-16T11:52:10.728 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94656.log 2023-12-16T11:52:10.728 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138609.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138609.log.gz 2023-12-16T11:52:10.728 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109924.log 2023-12-16T11:52:10.730 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.94656.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94656.log.gz 2023-12-16T11:52:10.730 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102255.log 2023-12-16T11:52:10.731 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109924.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109924.log.gz 2023-12-16T11:52:10.731 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111633.log 2023-12-16T11:52:10.732 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102255.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102255.log.gz 2023-12-16T11:52:10.732 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107619.log 2023-12-16T11:52:10.734 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111633.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111633.log.gz 2023-12-16T11:52:10.734 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118933.log 2023-12-16T11:52:10.736 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107619.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107619.log.gz 2023-12-16T11:52:10.736 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128703.log 2023-12-16T11:52:10.738 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118933.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118933.log.gz 2023-12-16T11:52:10.738 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98247.log 2023-12-16T11:52:10.738 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.128703.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128703.log.gz 2023-12-16T11:52:10.739 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103455.log 2023-12-16T11:52:10.741 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98247.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98247.log.gz 2023-12-16T11:52:10.741 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121010.log 2023-12-16T11:52:10.742 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103455.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103455.log.gz 2023-12-16T11:52:10.743 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108749.log 2023-12-16T11:52:10.743 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121010.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121010.log.gz 2023-12-16T11:52:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101202.log 2023-12-16T11:52:10.744 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108749.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108749.log.gz 2023-12-16T11:52:10.745 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108102.log 2023-12-16T11:52:10.746 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101202.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101202.log.gz 2023-12-16T11:52:10.747 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116488.log 2023-12-16T11:52:10.748 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108102.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108102.log.gz 2023-12-16T11:52:10.748 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106718.log 2023-12-16T11:52:10.749 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116488.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102700.log.gz 2023-12-16T11:52:10.750 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95800.log 2023-12-16T11:52:10.750 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106718.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116488.log.gz 2023-12-16T11:52:10.750 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136551.log 2023-12-16T11:52:10.758 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95800.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106718.log.gz 2023-12-16T11:52:10.758 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101301.log 2023-12-16T11:52:10.758 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136551.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136551.log.gz 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102053.log 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101301.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101301.log.gz 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135879.log 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102053.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102053.log.gz 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110701.log 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135879.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135879.log.gz 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117046.log 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110701.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110701.log.gz 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94172.log 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.117046.log: 87.4% 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117046.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.95800.log.gz 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100157.log 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117805.log 2023-12-16T11:52:10.759 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.94172.log: /var/log/ceph/cluster2-client.mirror.100157.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94172.log.gz 2023-12-16T11:52:10.779 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122585.log 2023-12-16T11:52:10.779 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.100157.log.gz 2023-12-16T11:52:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97460.log 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122585.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122585.log.gz 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117805.log.gz 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98009.log 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93483.log 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97460.log.gz 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128126.log 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.93483.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98009.log.gz 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134703.log 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.93483.log.gz 2023-12-16T11:52:10.780 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108283.log 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128126.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134703.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134703.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132432.log 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134199.log 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108283.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.132432.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132432.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129590.log 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134199.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139911.log 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.134199.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115254.log 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139911.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105824.log 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115254.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115254.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129590.log.gz 2023-12-16T11:52:10.781 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110524.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116727.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110524.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105824.log.gz 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110524.log.gz 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107751.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126455.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107751.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116727.log.gz 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107751.log.gz 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93773.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128188.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126455.log.gz 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.93773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107820.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128188.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.93773.log.gz 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128188.log.gz 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112159.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116956.log 2023-12-16T11:52:10.782 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112159.log: /var/log/ceph/cluster2-client.mirror.107820.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112159.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129488.log 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116956.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116956.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107820.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99791.log 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85929.log 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129488.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137601.log 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.85929.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85929.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.99791.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94924.log 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137601.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107075.log 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.137601.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125347.log 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107075.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94924.log.gz 2023-12-16T11:52:10.783 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107075.log.gz 2023-12-16T11:52:10.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137937.log 2023-12-16T11:52:10.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94876.log 2023-12-16T11:52:10.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125347.log: /var/log/ceph/cluster2-client.admin.137937.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125347.log.gz 2023-12-16T11:52:10.784 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137937.log.gz 2023-12-16T11:52:10.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102599.log 2023-12-16T11:52:10.784 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118518.log 2023-12-16T11:52:10.784 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94876.log: /var/log/ceph/cluster2-client.mirror.102599.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94876.log.gz 2023-12-16T11:52:10.801 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102599.log.gz 2023-12-16T11:52:10.801 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103036.log 2023-12-16T11:52:10.801 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96660.log 2023-12-16T11:52:10.801 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103036.log: /var/log/ceph/cluster2-client.mirror.118518.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103036.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113545.log 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.96660.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.96660.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118518.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117070.log 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.91766.log 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.117070.log: -- replaced with /var/log/ceph/cluster1-client.mirror.113545.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.117070.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120695.log 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105595.log 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.2.91766.log: /var/log/ceph/cluster1-client.mirror.120695.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120695.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108701.log 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105595.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105595.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106005.log 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.108701.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108701.log.gz 2023-12-16T11:52:10.802 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96217.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106005.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129050.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126334.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106005.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114054.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126334.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96217.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126334.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114755.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122270.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114755.log: 88.3% 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114054.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.91766.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106846.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.114755.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119759.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122270.log.gz 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91260.log 2023-12-16T11:52:10.803 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106846.log: /var/log/ceph/cluster2-client.mirror.119759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128167.log 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.91260.log: 87.3% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91260.log.gz 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.106846.log.gz 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108681.log 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111143.log 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119759.log.gz 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.108681.log: -- replaced with /var/log/ceph/cluster2-client.mirror.128167.log.gz 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135963.log 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112017.log 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135963.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108681.log.gz 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111143.log.gz 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135963.log.gz 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:10.804 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131592.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113783.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr: 88.4%/var/log/ceph/cluster2-client.admin.131592.log: -- replaced with /var/log/ceph/cluster2-client.mirror.112017.log.gz 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131592.log.gz 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112204.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97703.log/var/log/ceph/cluster2-client.mirror.113783.log: 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97236.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97703.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.113783.log.gz 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112204.log.gz 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120173.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121640.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97236.log: /var/log/ceph/cluster2-client.mirror.120173.log: 87.7% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97236.log.gz 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.120173.log.gz 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105035.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114717.log 2023-12-16T11:52:10.805 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121640.log: /var/log/ceph/cluster2-client.mirror.105035.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121640.log.gz 2023-12-16T11:52:10.806 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97273.log 2023-12-16T11:52:10.806 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105035.log.gz 2023-12-16T11:52:10.806 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.114717.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114717.log.gz 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117743.log 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97703.log.gz 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip/var/log/ceph/cluster1-client.mirror.97273.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95449.log 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104250.log 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.95449.log: -- replaced with /var/log/ceph/cluster2-client.mirror.117743.log.gz 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95449.log.gz 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97039.log 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117516.log 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97273.log.gz/var/log/ceph/cluster2-client.mirror.104250.log: 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102362.log 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104250.log.gz 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93749.log 2023-12-16T11:52:10.819 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117516.log.gz/var/log/ceph/cluster2-client.mirror.102362.log: 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103897.log 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.93749.log: -- replaced with /var/log/ceph/cluster2-client.mirror.102362.log.gz 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128615.log 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.93749.log.gz 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105077.log 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.128615.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103897.log.gz 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128615.log.gz 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127958.log 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110476.log 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105077.log.gz 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.127958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95517.log 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110476.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127958.log.gz 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110476.log.gz 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138819.log 2023-12-16T11:52:10.820 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4%/var/log/ceph/cluster1-client.admin.95517.log: -- replaced with /var/log/ceph/cluster2-client.mirror.97039.log.gz 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96555.log 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.95517.log.gz 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138819.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138819.log.gz 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98809.log 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93633.log 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119677.log 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.93633.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98809.log.gz 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93633.log.gz 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128950.log 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92212.log 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119677.log: /var/log/ceph/cluster2-client.mirror.128950.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119677.log.gz 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128950.log.gz 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114220.log 2023-12-16T11:52:10.821 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.92212.log: 52.7% -- replaced with /var/log/ceph/cluster1-client.mirror.92212.log.gz 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138735.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103712.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114220.log: /var/log/ceph/cluster2-client.admin.138735.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138735.log.gz 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114220.log.gz 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122607.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99951.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103712.log: /var/log/ceph/cluster1-client.mirror.122607.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103712.log.gz 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122607.log.gz 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103438.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127042.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99951.log: /var/log/ceph/cluster2-client.mirror.103438.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103438.log.gz 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.99951.log.gz 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113100.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121055.log 2023-12-16T11:52:10.822 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127042.log: /var/log/ceph/cluster2-client.mirror.113100.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127042.log.gz 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117660.log 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121055.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121055.log.gz 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119367.log 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.113100.log.gz/var/log/ceph/cluster2-client.mirror.117660.log: 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117660.log.gz 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.132768.log 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120317.log 2023-12-16T11:52:10.825 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119367.log: /var/log/ceph/cluster2-client.admin.132768.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.132768.log.gz 87.9% 2023-12-16T11:52:10.826 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119367.log.gz 2023-12-16T11:52:10.826 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124841.log 2023-12-16T11:52:10.826 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101109.log 2023-12-16T11:52:10.826 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120317.log: /var/log/ceph/cluster1-client.mirror.124841.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124841.log.gz 2023-12-16T11:52:10.826 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120317.log.gz 2023-12-16T11:52:10.826 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114912.log 2023-12-16T11:52:10.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100137.log 2023-12-16T11:52:10.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101109.log: /var/log/ceph/cluster1-client.mirror.114912.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101109.log.gz 2023-12-16T11:52:10.827 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91302.log 2023-12-16T11:52:10.827 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100137.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.100137.log.gz 2023-12-16T11:52:10.828 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111302.log 2023-12-16T11:52:10.828 INFO:teuthology.orchestra.run.smithi160.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114912.log.gz 2023-12-16T11:52:10.828 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.91302.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91302.log.gz 2023-12-16T11:52:10.828 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120379.log 2023-12-16T11:52:10.829 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111302.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111302.log.gz 2023-12-16T11:52:10.829 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122000.log 2023-12-16T11:52:10.830 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110918.log 2023-12-16T11:52:10.830 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120379.log: /var/log/ceph/cluster1-client.mirror.122000.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122000.log.gz 2023-12-16T11:52:10.831 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120379.log.gz 2023-12-16T11:52:10.831 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116568.log 2023-12-16T11:52:10.831 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128638.log 2023-12-16T11:52:10.832 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110918.log: /var/log/ceph/cluster2-client.mirror.116568.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110918.log.gz 2023-12-16T11:52:10.832 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.116568.log.gz 2023-12-16T11:52:10.832 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103554.log 2023-12-16T11:52:10.833 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.128638.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128638.log.gz 2023-12-16T11:52:10.834 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117536.log 2023-12-16T11:52:10.834 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103919.log 2023-12-16T11:52:10.834 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103554.log: /var/log/ceph/cluster2-client.mirror.117536.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.103554.log.gz 2023-12-16T11:52:10.835 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108355.log 2023-12-16T11:52:10.835 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103919.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117536.log.gz 2023-12-16T11:52:10.847 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90501.log 2023-12-16T11:52:10.848 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108355.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108355.log.gz 2023-12-16T11:52:10.860 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102542.log 2023-12-16T11:52:10.861 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90501.log.gz 2023-12-16T11:52:10.873 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96555.log.gz 2023-12-16T11:52:10.873 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100500.log 2023-12-16T11:52:10.873 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136089.log 2023-12-16T11:52:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92572.log 2023-12-16T11:52:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136089.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136089.log.gz 2023-12-16T11:52:10.896 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113738.log 2023-12-16T11:52:10.896 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.92572.log: 66.7% -- replaced with /var/log/ceph/cluster2-client.admin.92572.log.gz 2023-12-16T11:52:10.908 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129618.log 2023-12-16T11:52:10.909 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113738.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113738.log.gz 2023-12-16T11:52:10.919 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128909.log 2023-12-16T11:52:10.919 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.129618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129618.log.gz 2023-12-16T11:52:10.928 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104191.log.gz 2023-12-16T11:52:10.930 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109685.log 2023-12-16T11:52:10.932 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111123.log 2023-12-16T11:52:10.932 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128909.log.gz 2023-12-16T11:52:10.932 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109685.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109685.log.gz 2023-12-16T11:52:10.933 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106698.log 2023-12-16T11:52:10.934 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111123.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111123.log.gz 2023-12-16T11:52:10.934 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106133.log 2023-12-16T11:52:10.937 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106698.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106698.log.gz 2023-12-16T11:52:10.937 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111481.log 2023-12-16T11:52:10.938 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106133.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106133.log.gz 2023-12-16T11:52:10.938 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131172.log 2023-12-16T11:52:10.940 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111481.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111481.log.gz 2023-12-16T11:52:10.940 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103411.log 2023-12-16T11:52:10.941 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131172.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131172.log.gz 2023-12-16T11:52:10.941 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125370.log 2023-12-16T11:52:10.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108970.log 2023-12-16T11:52:10.955 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125370.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125370.log.gz 2023-12-16T11:52:10.965 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104820.log.gz 2023-12-16T11:52:10.970 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98582.log 2023-12-16T11:52:10.970 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122360.log 2023-12-16T11:52:10.970 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108970.log.gz 2023-12-16T11:52:10.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98582.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.98582.log.gz 2023-12-16T11:52:10.987 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129447.log 2023-12-16T11:52:10.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122360.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122360.log.gz 2023-12-16T11:52:10.987 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102520.log 2023-12-16T11:52:10.987 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129447.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129447.log.gz 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115210.log 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102520.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102520.log.gz 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130584.log 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115210.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115210.log.gz 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134535.log 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130584.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130584.log.gz 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97210.log 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134535.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134535.log.gz 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111388.log 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97210.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97210.log.gz 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98220.log 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111388.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111388.log.gz 2023-12-16T11:52:10.988 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117159.log 2023-12-16T11:52:10.989 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.98220.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.98220.log.gz 2023-12-16T11:52:10.989 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122495.log 2023-12-16T11:52:10.992 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.117159.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.117159.log.gz 2023-12-16T11:52:10.992 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102562.log 2023-12-16T11:52:10.992 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122495.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122495.log.gz 2023-12-16T11:52:10.992 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112921.log 2023-12-16T11:52:10.992 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102562.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102562.log.gz 2023-12-16T11:52:10.992 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136971.log 2023-12-16T11:52:10.993 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112921.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.112921.log.gz 2023-12-16T11:52:10.996 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106202.log 2023-12-16T11:52:10.996 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136971.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136971.log.gz 2023-12-16T11:52:10.996 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115973.log 2023-12-16T11:52:10.997 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106202.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106202.log.gz 2023-12-16T11:52:10.997 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114437.log 2023-12-16T11:52:10.998 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115973.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.115973.log.gz 2023-12-16T11:52:10.998 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122045.log 2023-12-16T11:52:10.998 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114437.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114437.log.gz 2023-12-16T11:52:11.000 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105114.log 2023-12-16T11:52:11.000 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122045.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122045.log.gz 2023-12-16T11:52:11.000 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110749.log 2023-12-16T11:52:11.002 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105114.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105114.log.gz 2023-12-16T11:52:11.002 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101244.log 2023-12-16T11:52:11.003 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110749.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110749.log.gz 2023-12-16T11:52:11.003 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109904.log 2023-12-16T11:52:11.004 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101244.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103919.log.gz 2023-12-16T11:52:11.005 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116468.log 2023-12-16T11:52:11.007 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109904.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109904.log.gz 2023-12-16T11:52:11.007 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117598.log 2023-12-16T11:52:11.009 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.116468.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116468.log.gz 2023-12-16T11:52:11.009 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118602.log 2023-12-16T11:52:11.013 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117598.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117598.log.gz 2023-12-16T11:52:11.014 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127909.log 2023-12-16T11:52:11.014 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118602.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118602.log.gz 2023-12-16T11:52:11.014 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102149.log 2023-12-16T11:52:11.026 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127645.log 2023-12-16T11:52:11.032 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138525.log 2023-12-16T11:52:11.043 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135459.log 2023-12-16T11:52:11.043 INFO:teuthology.orchestra.run.smithi160.stderr: 87.8%/var/log/ceph/cluster2-client.admin.138525.log: -- replaced with /var/log/ceph/cluster2-client.mirror.127645.log.gz 2023-12-16T11:52:11.043 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138525.log.gz 2023-12-16T11:52:11.051 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103095.log.gz 2023-12-16T11:52:11.052 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101991.log 2023-12-16T11:52:11.052 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135459.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135459.log.gz 2023-12-16T11:52:11.056 INFO:teuthology.orchestra.run.smithi160.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102149.log.gz 2023-12-16T11:52:11.067 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125067.log 2023-12-16T11:52:11.068 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101991.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101991.log.gz 2023-12-16T11:52:11.068 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126190.log 2023-12-16T11:52:11.068 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125067.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125067.log.gz 2023-12-16T11:52:11.068 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.128514.log 2023-12-16T11:52:11.068 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126190.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126190.log.gz 2023-12-16T11:52:11.069 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120852.log 2023-12-16T11:52:11.069 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.0.128514.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.128514.log.gz 2023-12-16T11:52:11.069 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.109786.log 2023-12-16T11:52:11.069 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120852.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120852.log.gz 2023-12-16T11:52:11.069 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111814.log 2023-12-16T11:52:11.082 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.109786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101577.log 2023-12-16T11:52:11.082 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111814.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111814.log.gz 2023-12-16T11:52:11.098 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105057.log 2023-12-16T11:52:11.099 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101577.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101577.log.gz 2023-12-16T11:52:11.099 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103411.log.gz 2023-12-16T11:52:11.099 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115614.log 2023-12-16T11:52:11.103 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105057.log: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.109786.log.gz 2023-12-16T11:52:11.104 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121865.log 2023-12-16T11:52:11.105 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115614.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115614.log.gz 2023-12-16T11:52:11.106 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114148.log 2023-12-16T11:52:11.106 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121865.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121865.log.gz 2023-12-16T11:52:11.106 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124399.log 2023-12-16T11:52:11.107 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114148.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114148.log.gz 2023-12-16T11:52:11.107 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113566.log 2023-12-16T11:52:11.107 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124399.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124399.log.gz 2023-12-16T11:52:11.108 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118291.log 2023-12-16T11:52:11.110 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113566.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.113566.log.gz 2023-12-16T11:52:11.110 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100768.log 2023-12-16T11:52:11.111 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118291.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118291.log.gz 2023-12-16T11:52:11.111 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133356.log 2023-12-16T11:52:11.112 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100768.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100768.log.gz 2023-12-16T11:52:11.112 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118953.log 2023-12-16T11:52:11.113 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133356.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133356.log.gz 2023-12-16T11:52:11.113 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.93873.log 2023-12-16T11:52:11.115 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118953.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118953.log.gz 2023-12-16T11:52:11.115 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80930.log 2023-12-16T11:52:11.120 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.93873.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.93873.log.gz 2023-12-16T11:52:11.121 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119573.log 2023-12-16T11:52:11.121 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph.tmp-client.admin.80930.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80930.log.gz 2023-12-16T11:52:11.121 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99023.log 2023-12-16T11:52:11.123 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119573.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119573.log.gz 2023-12-16T11:52:11.123 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139365.log 2023-12-16T11:52:11.124 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99023.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100500.log.gz 2023-12-16T11:52:11.130 INFO:teuthology.orchestra.run.smithi160.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99023.log.gz 2023-12-16T11:52:11.131 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94120.log 2023-12-16T11:52:11.131 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98421.log 2023-12-16T11:52:11.131 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139365.log: /var/log/ceph/cluster1-client.admin.94120.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139365.log.gz 2023-12-16T11:52:11.131 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121212.log 2023-12-16T11:52:11.131 INFO:teuthology.orchestra.run.smithi160.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94120.log.gz 2023-12-16T11:52:11.134 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.98421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92255.log 2023-12-16T11:52:11.134 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121212.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121212.log.gz 2023-12-16T11:52:11.134 INFO:teuthology.orchestra.run.smithi160.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98421.log.gz 2023-12-16T11:52:11.134 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.94693.log 2023-12-16T11:52:11.134 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.92255.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.92255.log.gzgzip 2023-12-16T11:52:11.134 INFO:teuthology.orchestra.run.smithi160.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128371.log 2023-12-16T11:52:11.134 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124461.log 2023-12-16T11:52:11.138 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128371.log: /var/log/ceph/cluster1-client.mirror.0.94693.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.94693.log.gz 2023-12-16T11:52:11.139 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128106.log 2023-12-16T11:52:11.139 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124461.log: 86.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124461.log.gz 2023-12-16T11:52:11.140 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121280.log 2023-12-16T11:52:11.140 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128106.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128106.log.gz 2023-12-16T11:52:11.141 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102483.log 2023-12-16T11:52:11.141 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121280.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121280.log.gz 2023-12-16T11:52:11.142 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136887.log 2023-12-16T11:52:11.142 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102483.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102483.log.gz 2023-12-16T11:52:11.143 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90502.log 2023-12-16T11:52:11.143 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136887.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136887.log.gz 2023-12-16T11:52:11.143 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138945.log 2023-12-16T11:52:11.144 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90502.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90502.log.gz 2023-12-16T11:52:11.144 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138147.log 2023-12-16T11:52:11.144 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138945.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138945.log.gz 2023-12-16T11:52:11.145 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124314.log 2023-12-16T11:52:11.146 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138147.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138147.log.gz 2023-12-16T11:52:11.146 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94448.log 2023-12-16T11:52:11.146 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.124314.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124314.log.gz 2023-12-16T11:52:11.146 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122771.log 2023-12-16T11:52:11.147 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94448.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.94448.log.gz 2023-12-16T11:52:11.148 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128514.log 2023-12-16T11:52:11.148 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.122771.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122771.log.gz 2023-12-16T11:52:11.149 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104149.log 2023-12-16T11:52:11.149 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.128514.log: 79.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128514.log.gz 2023-12-16T11:52:11.149 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112364.log 2023-12-16T11:52:11.150 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104149.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104149.log.gz 2023-12-16T11:52:11.151 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105193.log 2023-12-16T11:52:11.151 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.112364.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112364.log.gz 2023-12-16T11:52:11.152 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117247.log 2023-12-16T11:52:11.152 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105193.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105193.log.gz 2023-12-16T11:52:11.153 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104899.log 2023-12-16T11:52:11.153 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117247.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117247.log.gz 2023-12-16T11:52:11.154 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139323.log 2023-12-16T11:52:11.167 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127021.log 2023-12-16T11:52:11.168 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139323.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139323.log.gz 2023-12-16T11:52:11.182 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135417.log 2023-12-16T11:52:11.189 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127021.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126392.log 2023-12-16T11:52:11.189 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.127021.log.gz 2023-12-16T11:52:11.196 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101848.log 2023-12-16T11:52:11.196 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135417.log.gz 2023-12-16T11:52:11.202 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126392.log: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93677.log 2023-12-16T11:52:11.202 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.126392.log.gz 2023-12-16T11:52:11.209 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131970.log 2023-12-16T11:52:11.220 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.93677.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93677.log.gz 2023-12-16T11:52:11.222 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101343.log 2023-12-16T11:52:11.222 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131970.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131970.log.gz 2023-12-16T11:52:11.231 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100028.log 2023-12-16T11:52:11.231 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101343.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.101343.log.gz 2023-12-16T11:52:11.240 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102621.log 2023-12-16T11:52:11.253 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116174.log 2023-12-16T11:52:11.262 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109706.log 2023-12-16T11:52:11.271 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118167.log 2023-12-16T11:52:11.275 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.109706.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.116174.log.gz 2023-12-16T11:52:11.279 INFO:teuthology.orchestra.run.smithi160.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100028.log.gz 2023-12-16T11:52:11.285 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109706.log.gz 2023-12-16T11:52:11.291 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101514.log 2023-12-16T11:52:11.292 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118167.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118167.log.gz 2023-12-16T11:52:11.299 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101244.log.gz 2023-12-16T11:52:11.311 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101789.log 2023-12-16T11:52:11.312 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101514.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.101514.log.gz 2023-12-16T11:52:11.314 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102542.log.gz 2023-12-16T11:52:11.320 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117784.log 2023-12-16T11:52:11.333 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101789.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.101789.log.gz 2023-12-16T11:52:11.338 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99871.log 2023-12-16T11:52:11.340 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117784.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117784.log.gz 2023-12-16T11:52:11.341 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.105057.log.gz 2023-12-16T11:52:11.342 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94376.log 2023-12-16T11:52:11.342 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99871.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99871.log.gz 2023-12-16T11:52:11.342 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99605.log 2023-12-16T11:52:11.344 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.94376.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94376.log.gz 2023-12-16T11:52:11.344 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.138651.log 2023-12-16T11:52:11.344 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99605.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104899.log.gz 2023-12-16T11:52:11.349 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99605.log.gz 2023-12-16T11:52:11.351 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.138651.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.138651.log.gz 2023-12-16T11:52:11.351 INFO:teuthology.orchestra.run.smithi160.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.99304.log 2023-12-16T11:52:11.351 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103998.log 2023-12-16T11:52:11.351 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139407.log 2023-12-16T11:52:11.356 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103998.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99304.log.gz 2023-12-16T11:52:11.357 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127909.log.gz 2023-12-16T11:52:11.359 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118332.log 2023-12-16T11:52:11.359 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.139407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103073.log 2023-12-16T11:52:11.359 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.139407.log.gz 2023-12-16T11:52:11.360 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118332.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118332.log.gz 2023-12-16T11:52:11.362 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140415.log 2023-12-16T11:52:11.362 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110942.log 2023-12-16T11:52:11.362 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103073.log.gz 2023-12-16T11:52:11.362 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.140415.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140415.log.gz 2023-12-16T11:52:11.362 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99851.log 2023-12-16T11:52:11.364 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.110942.log: -- /var/log/ceph/cluster1-client.mirror.1.105444.log 2023-12-16T11:52:11.364 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110942.log.gz 2023-12-16T11:52:11.364 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99851.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99851.log.gz 2023-12-16T11:52:11.364 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102011.log 2023-12-16T11:52:11.364 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.105444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97059.log 2023-12-16T11:52:11.365 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102011.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.102011.log.gz 2023-12-16T11:52:11.365 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124965.log 2023-12-16T11:52:11.370 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.97059.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97059.log.gz 2023-12-16T11:52:11.371 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117474.log 2023-12-16T11:52:11.371 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124965.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124965.log.gz 2023-12-16T11:52:11.372 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136929.log 2023-12-16T11:52:11.373 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117474.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117474.log.gz 2023-12-16T11:52:11.374 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122067.log 2023-12-16T11:52:11.374 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136929.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136929.log.gz 2023-12-16T11:52:11.374 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91766.log 2023-12-16T11:52:11.375 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122067.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122067.log.gz 2023-12-16T11:52:11.376 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119325.log 2023-12-16T11:52:11.376 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.91766.log: 73.6% -- replaced with /var/log/ceph/cluster1-client.mirror.91766.log.gz 2023-12-16T11:52:11.377 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118891.log 2023-12-16T11:52:11.379 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119325.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119325.log.gz 2023-12-16T11:52:11.379 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101072.log 2023-12-16T11:52:11.380 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118891.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118891.log.gz 2023-12-16T11:52:11.382 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106561.log 2023-12-16T11:52:11.382 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101072.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101072.log.gz 2023-12-16T11:52:11.382 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115551.log 2023-12-16T11:52:11.383 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106561.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106561.log.gz 2023-12-16T11:52:11.384 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107123.log 2023-12-16T11:52:11.385 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115551.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115551.log.gz 2023-12-16T11:52:11.385 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125943.log 2023-12-16T11:52:11.386 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107123.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107123.log.gz 2023-12-16T11:52:11.386 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120627.log 2023-12-16T11:52:11.397 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125943.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.105444.log.gz 2023-12-16T11:52:11.399 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115141.log 2023-12-16T11:52:11.400 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120627.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120627.log.gz 2023-12-16T11:52:11.401 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.101848.log.gz 2023-12-16T11:52:11.413 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128044.log 2023-12-16T11:52:11.413 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128888.log 2023-12-16T11:52:11.414 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115141.log: /var/log/ceph/cluster2-client.mirror.128044.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128044.log.gz 2023-12-16T11:52:11.415 INFO:teuthology.orchestra.run.smithi160.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115141.log.gz 2023-12-16T11:52:11.415 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131718.log 2023-12-16T11:52:11.418 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130752.log 2023-12-16T11:52:11.418 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128888.log: /var/log/ceph/cluster2-client.admin.131718.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131718.log.gz 2023-12-16T11:52:11.418 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128888.log.gz 2023-12-16T11:52:11.418 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120131.log 2023-12-16T11:52:11.418 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130752.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93814.log 2023-12-16T11:52:11.418 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.130752.log.gz 2023-12-16T11:52:11.418 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.120131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112537.log 2023-12-16T11:52:11.419 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.93814.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120131.log.gz 2023-12-16T11:52:11.423 INFO:teuthology.orchestra.run.smithi160.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.93814.log.gz 2023-12-16T11:52:11.425 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118250.log 2023-12-16T11:52:11.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105156.log 2023-12-16T11:52:11.425 INFO:teuthology.orchestra.run.smithi160.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112537.log.gz 2023-12-16T11:52:11.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118250.log: gzip -5 --verbose 87.7% -- /var/log/ceph/cluster1-client.mirror.110572.log 2023-12-16T11:52:11.425 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.118250.log.gz 2023-12-16T11:52:11.425 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.105156.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105156.log.gz 2023-12-16T11:52:11.427 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96922.log 2023-12-16T11:52:11.428 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110572.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104213.log 2023-12-16T11:52:11.428 INFO:teuthology.orchestra.run.smithi160.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110572.log.gz/var/log/ceph/cluster1-client.mirror.96922.log: 2023-12-16T11:52:11.428 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99261.log 2023-12-16T11:52:11.428 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104213.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.96922.log.gz 2023-12-16T11:52:11.428 INFO:teuthology.orchestra.run.smithi160.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.104213.log.gz 2023-12-16T11:52:11.428 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134241.log 2023-12-16T11:52:11.432 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.99261.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99261.log.gz 2023-12-16T11:52:11.433 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104877.log 2023-12-16T11:52:11.433 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115347.log 2023-12-16T11:52:11.433 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.134241.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134241.log.gz 2023-12-16T11:52:11.434 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112762.log 2023-12-16T11:52:11.434 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104877.log.gz 2023-12-16T11:52:11.435 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.115347.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115347.log.gz 2023-12-16T11:52:11.436 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99347.log 2023-12-16T11:52:11.436 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.90900.log 2023-12-16T11:52:11.437 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.112762.log: /var/log/ceph/cluster2-client.mirror.99347.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112762.log.gz 2023-12-16T11:52:11.437 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129385.log 2023-12-16T11:52:11.437 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.90900.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.90900.log.gz 2023-12-16T11:52:11.438 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135207.log 2023-12-16T11:52:11.438 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.129385.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129385.log.gz 2023-12-16T11:52:11.439 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136719.log 2023-12-16T11:52:11.439 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135207.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135207.log.gz 2023-12-16T11:52:11.439 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103497.log 2023-12-16T11:52:11.440 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136719.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136719.log.gz 2023-12-16T11:52:11.440 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125393.log 2023-12-16T11:52:11.442 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103497.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.99347.log.gz 2023-12-16T11:52:11.442 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102621.log.gz 2023-12-16T11:52:11.457 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.125393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120672.log 2023-12-16T11:52:11.457 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125393.log.gz 2023-12-16T11:52:11.457 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121460.log 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126912.log 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120672.log.gz 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.137349.log 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126912.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121460.log.gz 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95759.log 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126912.log.gz 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.137349.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.137349.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130206.log 2023-12-16T11:52:11.458 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135039.log 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95759.log.gz 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130206.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130206.log.gz 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102238.log 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135039.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118850.log 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.135039.log.gz 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.102238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118270.log 2023-12-16T11:52:11.459 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118850.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.102238.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111364.log 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118850.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130416.log 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.111364.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111364.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120605.log 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130416.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130416.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118270.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115696.log 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.120605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120605.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.91008.log 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-osd.0.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115696.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106085.log 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.91008.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.91008.log.gz 2023-12-16T11:52:11.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106629.log 2023-12-16T11:52:11.462 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.106085.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106085.log.gz 2023-12-16T11:52:11.462 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95843.log 2023-12-16T11:52:11.462 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.106629.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106629.log.gz 2023-12-16T11:52:11.462 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126499.log 2023-12-16T11:52:11.465 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.95843.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95843.log.gz 2023-12-16T11:52:11.465 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113803.log 2023-12-16T11:52:11.466 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.126499.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126499.log.gz 2023-12-16T11:52:11.466 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108307.log 2023-12-16T11:52:11.468 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.113803.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113803.log.gz 2023-12-16T11:52:11.468 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.133188.log 2023-12-16T11:52:11.469 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108307.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108307.log.gz 2023-12-16T11:52:11.469 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95364.log 2023-12-16T11:52:11.470 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.133188.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.133188.log.gz 2023-12-16T11:52:11.471 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119925.log 2023-12-16T11:52:11.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95364.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95364.log.gz 2023-12-16T11:52:11.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92555.log 2023-12-16T11:52:11.477 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119925.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119925.log.gz 2023-12-16T11:52:11.477 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122716.log 2023-12-16T11:52:11.478 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.92555.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.92555.log.gz 2023-12-16T11:52:11.478 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98561.log 2023-12-16T11:52:11.478 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.122716.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122716.log.gz 2023-12-16T11:52:11.479 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126935.log 2023-12-16T11:52:11.479 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.98561.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98561.log.gz 2023-12-16T11:52:11.479 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92657.log 2023-12-16T11:52:11.480 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126935.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126935.log.gz 2023-12-16T11:52:11.482 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96878.log 2023-12-16T11:52:11.483 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.92657.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92657.log.gz 2023-12-16T11:52:11.483 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121752.log 2023-12-16T11:52:11.483 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.96878.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96878.log.gz 2023-12-16T11:52:11.484 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115503.log 2023-12-16T11:52:11.484 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121752.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121752.log.gz 2023-12-16T11:52:11.484 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110388.log 2023-12-16T11:52:11.484 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115503.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115503.log.gz 2023-12-16T11:52:11.485 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107051.log 2023-12-16T11:52:11.486 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110388.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110388.log.gz 2023-12-16T11:52:11.487 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127083.log 2023-12-16T11:52:11.488 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.107051.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107051.log.gz 2023-12-16T11:52:11.488 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116893.log 2023-12-16T11:52:11.489 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127083.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127083.log.gz 2023-12-16T11:52:11.489 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122382.log 2023-12-16T11:52:11.490 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116893.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.116893.log.gz 2023-12-16T11:52:11.490 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124523.log 2023-12-16T11:52:11.491 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.122382.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122382.log.gz 2023-12-16T11:52:11.491 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116640.log 2023-12-16T11:52:11.492 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124523.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124523.log.gz 2023-12-16T11:52:11.492 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99281.log 2023-12-16T11:52:11.493 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.116640.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.116640.log.gz 2023-12-16T11:52:11.493 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130038.log 2023-12-16T11:52:11.498 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.99281.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99281.log.gz 2023-12-16T11:52:11.499 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110966.log 2023-12-16T11:52:11.499 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.130038.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130038.log.gz 2023-12-16T11:52:11.500 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124293.log 2023-12-16T11:52:11.500 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.110966.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110966.log.gz 2023-12-16T11:52:11.501 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128827.log 2023-12-16T11:52:11.501 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124293.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124293.log.gz 2023-12-16T11:52:11.502 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110456.log 2023-12-16T11:52:11.504 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128827.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128827.log.gz 2023-12-16T11:52:11.504 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117022.log 2023-12-16T11:52:11.504 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.110456.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.110456.log.gz 2023-12-16T11:52:11.505 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109726.log 2023-12-16T11:52:11.505 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117022.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117022.log.gz 2023-12-16T11:52:11.506 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.136131.log 2023-12-16T11:52:11.507 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.109726.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109726.log.gz 2023-12-16T11:52:11.507 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117268.log 2023-12-16T11:52:11.508 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.136131.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.136131.log.gz 2023-12-16T11:52:11.508 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103976.log 2023-12-16T11:52:11.509 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117268.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117268.log.gz 2023-12-16T11:52:11.509 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124379.log 2023-12-16T11:52:11.510 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103976.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.103976.log.gz 2023-12-16T11:52:11.510 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118747.log 2023-12-16T11:52:11.511 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.124379.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124379.log.gz 2023-12-16T11:52:11.511 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101534.log 2023-12-16T11:52:11.512 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.118747.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118747.log.gz 2023-12-16T11:52:11.513 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121977.log 2023-12-16T11:52:11.513 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.101534.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.101534.log.gz 2023-12-16T11:52:11.513 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100810.log 2023-12-16T11:52:11.514 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121977.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121977.log.gz 2023-12-16T11:52:11.514 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121617.log 2023-12-16T11:52:11.528 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129764.log 2023-12-16T11:52:11.528 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.121617.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121617.log.gz 2023-12-16T11:52:11.541 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114799.log 2023-12-16T11:52:11.543 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.admin.129764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.129764.log.gz 2023-12-16T11:52:11.543 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119305.log 2023-12-16T11:52:11.543 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.114799.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114799.log.gz 2023-12-16T11:52:11.556 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97937.log 2023-12-16T11:52:11.557 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.119305.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.119305.log.gz 2023-12-16T11:52:11.570 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126022.log 2023-12-16T11:52:11.571 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.97937.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97937.log.gz 2023-12-16T11:52:11.576 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108054.log 2023-12-16T11:52:11.577 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126022.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126022.log.gz 2023-12-16T11:52:11.586 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113646.log 2023-12-16T11:52:11.586 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108054.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108054.log.gz 2023-12-16T11:52:11.594 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103194.log 2023-12-16T11:52:11.608 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.113646.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113646.log.gz 2023-12-16T11:52:11.616 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104132.log 2023-12-16T11:52:11.616 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.103194.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103194.log.gz 2023-12-16T11:52:11.626 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125943.log.gz 2023-12-16T11:52:11.626 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108617.log 2023-12-16T11:52:11.627 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104132.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104132.log.gz 2023-12-16T11:52:11.628 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.102149.log 2023-12-16T11:52:11.630 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108617.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.108617.log.gz 2023-12-16T11:52:11.630 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108552.log 2023-12-16T11:52:11.644 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.1.102149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131298.log 2023-12-16T11:52:11.645 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.108552.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.108552.log.gz 2023-12-16T11:52:11.645 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103998.log.gz 2023-12-16T11:52:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127342.log 2023-12-16T11:52:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131298.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131298.log.gz 2023-12-16T11:52:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131634.log 2023-12-16T11:52:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127342.log: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127342.log.gz 2023-12-16T11:52:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100326.log 2023-12-16T11:52:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.131634.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131634.log.gz 2023-12-16T11:52:11.649 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128765.log 2023-12-16T11:52:11.651 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.100326.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.100326.log.gz 2023-12-16T11:52:11.651 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125235.log 2023-12-16T11:52:11.651 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.128765.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128765.log.gz 2023-12-16T11:52:11.651 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- 93.7% /var/log/ceph/cluster1-client.mirror.92195.log 2023-12-16T11:52:11.651 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.128371.log.gz 2023-12-16T11:52:11.654 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.125235.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125235.log.gz 2023-12-16T11:52:11.654 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117640.log 2023-12-16T11:52:11.654 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.92195.log: -- /var/log/ceph/cluster2-client.mirror.104956.log 2023-12-16T11:52:11.654 INFO:teuthology.orchestra.run.smithi160.stderr: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.92195.log.gz 2023-12-16T11:52:11.655 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.117640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127144.log 2023-12-16T11:52:11.655 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.104956.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117640.log.gz 2023-12-16T11:52:11.655 INFO:teuthology.orchestra.run.smithi160.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104956.log.gz 2023-12-16T11:52:11.655 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120193.log 2023-12-16T11:52:11.655 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95203.log 2023-12-16T11:52:11.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127144.log: /var/log/ceph/cluster2-client.mirror.120193.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127144.log.gz 2023-12-16T11:52:11.657 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111746.log 2023-12-16T11:52:11.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.95203.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120193.log.gz 2023-12-16T11:52:11.657 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107232.log 2023-12-16T11:52:11.657 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.111746.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.95203.log.gz 2023-12-16T11:52:11.657 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.91726.log 2023-12-16T11:52:11.659 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.107232.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107232.log.gz 2023-12-16T11:52:11.659 INFO:teuthology.orchestra.run.smithi160.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111746.log.gz 2023-12-16T11:52:11.659 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127185.log 2023-12-16T11:52:11.659 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.104035.log 2023-12-16T11:52:11.660 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.mirror.127185.log: /var/log/ceph/cluster2-client.mirror.1.91726.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.127185.log.gz 2023-12-16T11:52:11.660 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115302.log 2023-12-16T11:52:11.661 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.0.104035.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.91726.log.gz 2023-12-16T11:52:11.661 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135627.log 2023-12-16T11:52:11.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.115302.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.115302.log.gz 2023-12-16T11:52:11.663 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103497.log.gz 2023-12-16T11:52:11.663 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126087.log 2023-12-16T11:52:11.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster2-client.admin.135627.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135627.log.gz 2023-12-16T11:52:11.663 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cluster1-client.mirror.126087.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126087.log.gz 2023-12-16T11:52:11.669 INFO:teuthology.orchestra.run.smithi160.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.102149.log.gz 2023-12-16T11:52:11.732 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.100810.log.gz 2023-12-16T11:52:11.749 INFO:teuthology.orchestra.run.smithi160.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.104035.log.gz 2023-12-16T11:52:14.699 INFO:teuthology.orchestra.run.smithi087.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-12-16T11:52:15.327 INFO:teuthology.orchestra.run.smithi087.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-12-16T11:52:15.619 INFO:teuthology.orchestra.run.smithi087.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-12-16T11:52:15.621 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-12-16T11:52:15.621 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m8.003s 2023-12-16T11:52:15.621 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m23.457s 2023-12-16T11:52:15.622 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.936s 2023-12-16T11:52:54.558 INFO:teuthology.orchestra.run.smithi160.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-12-16T11:52:55.002 INFO:teuthology.orchestra.run.smithi160.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-12-16T11:53:10.198 INFO:teuthology.orchestra.run.smithi160.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-12-16T11:53:10.200 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-12-16T11:53:10.200 INFO:teuthology.orchestra.run.smithi160.stderr:real 1m2.547s 2023-12-16T11:53:10.200 INFO:teuthology.orchestra.run.smithi160.stderr:user 2m48.128s 2023-12-16T11:53:10.201 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m7.495s 2023-12-16T11:53:10.201 INFO:tasks.ceph:Archiving logs... 2023-12-16T11:53:10.201 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/remote/smithi087/log 2023-12-16T11:53:10.202 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-16T11:53:13.314 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/remote/smithi160/log 2023-12-16T11:53:13.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-16T11:53:38.510 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-16T11:53:38.523 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-16T11:53:38.524 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-16T11:53:38.525 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-16T11:53:38.527 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-16T11:53:38.606 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-16T11:53:38.618 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-16T11:53:38.618 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T11:53:38.621 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T11:53:38.634 INFO:teuthology.orchestra.run.smithi087.stderr:bash: line 1: ntpq: command not found 2023-12-16T11:53:38.637 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2023-12-16T11:53:38.638 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T11:53:38.638 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2023-12-16T11:53:38.639 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +304us[ +347us] +/- 68ms 2023-12-16T11:53:38.639 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 7 377 46 -670us[ -627us] +/- 66ms 2023-12-16T11:53:38.639 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +496us[ +539us] +/- 66ms 2023-12-16T11:53:38.639 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:53:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T11:53:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2023-12-16T11:53:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 178 +350us[ +392us] +/- 66ms 2023-12-16T11:53:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv02.front.sepia.ceph.com 3 8 377 52 -638us[ -595us] +/- 66ms 2023-12-16T11:53:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 9 377 185 +577us[ +619us] +/- 92ms 2023-12-16T11:53:38.642 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T11:53:38.643 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-16T11:53:38.656 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-16T11:53:38.657 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-16T11:53:38.671 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-16T11:53:38.776 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-16T11:53:38.928 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T11:53:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T11:53:38.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T11:53:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T11:53:38.995 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-16T11:53:39.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-16T11:53:39.040 INFO:teuthology.task.internal:Duration was 3169.603165 seconds 2023-12-16T11:53:39.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-16T11:53:39.056 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-16T11:53:39.056 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T11:53:39.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T11:53:39.099 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T11:53:39.101 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T11:53:39.515 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-16T11:53:39.515 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2023-12-16T11:53:39.515 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-16T11:53:39.536 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2023-12-16T11:53:39.536 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-16T11:53:39.576 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-16T11:53:39.576 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-16T11:53:39.578 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-16T11:53:39.645 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-16T11:53:39.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-16T11:53:39.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-16T11:53:39.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-16T11:53:39.803 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-16T11:53:39.804 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T11:53:39.831 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T11:53:39.858 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-16T11:53:39.872 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T11:53:39.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T11:53:39.897 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2023-12-16T11:53:39.923 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2023-12-16T11:53:39.939 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-16T11:53:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T11:53:39.981 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-16T11:53:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T11:53:39.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-16T11:53:40.009 INFO:teuthology.task.internal:Transferring archived files... 2023-12-16T11:53:40.010 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/remote/smithi087 2023-12-16T11:53:40.010 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T11:53:40.057 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988/remote/smithi160 2023-12-16T11:53:40.057 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T11:53:40.093 INFO:teuthology.task.internal:Removing archive directory... 2023-12-16T11:53:40.094 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-16T11:53:40.096 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-16T11:53:40.143 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-16T11:53:40.158 INFO:teuthology.task.internal:Not uploading archives. 2023-12-16T11:53:40.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-16T11:53:40.184 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-16T11:53:40.184 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T11:53:40.186 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T11:53:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 11:53 /home/ubuntu/cephtest 2023-12-16T11:53:40.200 INFO:teuthology.orchestra.run.smithi160.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 11:53 /home/ubuntu/cephtest 2023-12-16T11:53:40.202 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-16T11:53:40.218 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-16T11:53:40.243 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi160 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-16T11:53:40.244 INFO:teuthology.nuke:Checking targets against current locks 2023-12-16T11:53:40.269 DEBUG:teuthology.nuke:shortname: smithi087 2023-12-16T11:53:40.270 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T11:53:40.292 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 10:51:46.992645', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPkeJ1senFIzDxS3+j0Z32wtuNeGa/Dm2hwzGwxTXgWXVMGTT5Ri8kCA/tcuyDVlebct8GeVWwqpOqLgBW1Myw4='} 2023-12-16T11:53:40.295 DEBUG:teuthology.nuke:shortname: smithi160 2023-12-16T11:53:40.295 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T11:53:40.318 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:13:26-rbd-main-distro-default-smithi/7493988', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 10:51:46.991003', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjmBWJ/0sPWKHpp3zLV72klfKcofmL77fVFaUFRdBAiT3e1l9nCNfxDLV1niDNP2m9XRBvQQUBkDRhoK3L5IRQ='} 2023-12-16T11:53:40.348 INFO:teuthology.orchestra.console.smithi087:Power off 2023-12-16T11:53:40.348 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T11:53:40.372 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2023-12-16T11:53:40.372 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T11:53:40.387 INFO:teuthology.orchestra.console.smithi160:Power off 2023-12-16T11:53:40.388 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T11:53:40.396 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2023-12-16T11:53:40.410 DEBUG:teuthology.orchestra.console.smithi160:power off output: Chassis Power Control: Down/Off 2023-12-16T11:53:40.410 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T11:53:40.435 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-12-16T11:53:44.398 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T11:53:44.436 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T11:53:44.507 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2023-12-16T11:53:44.546 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-12-16T11:53:48.512 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T11:53:48.548 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T11:53:48.621 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2023-12-16T11:53:48.657 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is off 2023-12-16T11:53:48.723 INFO:teuthology.orchestra.console.smithi087:Power off completed 2023-12-16T11:53:48.758 INFO:teuthology.orchestra.console.smithi160:Power off completed 2023-12-16T11:53:48.864 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 3169.6031651496887 failure_reason: 'Command failed on smithi160 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json''' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-12-16T11:53:48.864 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T11:53:48.949 INFO:teuthology.run:FAIL